2026-02-25T21:50:30.876 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-25T21:50:30.889 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T21:50:30.961 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_21:45:26-rados:standalone-wip-mon-colocate-distro-default-trial/70636 branch: wip-mon-colocate description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: null first_in_suite: false flavor: default job_id: '70636' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: emzhang-2026-02-25_21:45:26-rados:standalone-wip-mon-colocate-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-mon-colocate ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd workunit: branch: wip-mon-colocate sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd owner: scheduled_emzhang@soko04.front.sepia.ceph.com priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9381 sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-mon-colocate suite_path: /home/teuthworker/src/github.com_Ericmzhang_ceph_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa suite_relpath: qa suite_repo: https://github.com/Ericmzhang/ceph.git suite_sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd targets: trial163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5z7FJOcE5GxEGrH76xlYupY7LTJF0Kmy1ulwLkMlFCHk56nTJ9ZbXr9J4cLUAHAvZGdh4KET/vypYsDJmdRuM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-25_21:45:26 tube: trial user: emzhang verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T21:50:30.961 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Ericmzhang_ceph_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa; will attempt to use it 2026-02-25T21:50:30.961 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Ericmzhang_ceph_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa/tasks 2026-02-25T21:50:30.961 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T21:50:30.963 INFO:teuthology.task.internal:Checking packages... 2026-02-25T21:50:30.963 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd' 2026-02-25T21:50:30.963 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T21:50:30.963 INFO:teuthology.packaging:ref: None 2026-02-25T21:50:30.963 INFO:teuthology.packaging:tag: None 2026-02-25T21:50:30.963 INFO:teuthology.packaging:branch: wip-mon-colocate 2026-02-25T21:50:30.963 INFO:teuthology.packaging:sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T21:50:30.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-mon-colocate 2026-02-25T21:50:31.081 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5305-g0886691e-1jammy 2026-02-25T21:50:31.082 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T21:50:31.093 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T21:50:31.093 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T21:50:31.103 INFO:teuthology.task.internal:Saving configuration 2026-02-25T21:50:31.110 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T21:50:31.121 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T21:50:31.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial163.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/emzhang-2026-02-25_21:45:26-rados:standalone-wip-mon-colocate-distro-default-trial/70636', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 21:46:55.899392', 'locked_by': 'scheduled_emzhang@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP0YfvstvCmI1T6Zfg/YvmOYVdHmcM2a+c+wKUkh17nW'} 2026-02-25T21:50:31.195 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T21:50:31.207 INFO:teuthology.task.internal:roles: ubuntu@trial163.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T21:50:31.207 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T21:50:31.282 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7458e42ee980>, signals=[15]) 2026-02-25T21:50:31.282 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T21:50:31.296 INFO:teuthology.task.internal:Opening connections... 2026-02-25T21:50:31.296 DEBUG:teuthology.task.internal:connecting to ubuntu@trial163.front.sepia.ceph.com 2026-02-25T21:50:31.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T21:50:31.360 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T21:50:31.372 DEBUG:teuthology.orchestra.run.trial163:> uname -m 2026-02-25T21:50:31.440 INFO:teuthology.orchestra.run.trial163.stdout:x86_64 2026-02-25T21:50:31.440 DEBUG:teuthology.orchestra.run.trial163:> cat /etc/os-release 2026-02-25T21:50:31.488 INFO:teuthology.orchestra.run.trial163.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:NAME="Ubuntu" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:VERSION_ID="22.04" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:VERSION_CODENAME=jammy 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:ID=ubuntu 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:ID_LIKE=debian 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-25T21:50:31.489 INFO:teuthology.orchestra.run.trial163.stdout:UBUNTU_CODENAME=jammy 2026-02-25T21:50:31.490 INFO:teuthology.lock.ops:Updating trial163.front.sepia.ceph.com on lock server 2026-02-25T21:50:31.586 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T21:50:31.599 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T21:50:31.612 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T21:50:31.612 DEBUG:teuthology.orchestra.run.trial163:> test '!' -e /home/ubuntu/cephtest 2026-02-25T21:50:31.616 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T21:50:31.629 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T21:50:31.629 DEBUG:teuthology.orchestra.run.trial163:> test -z $(ls -A /var/lib/ceph) 2026-02-25T21:50:31.661 INFO:teuthology.orchestra.run.trial163.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-25T21:50:31.662 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T21:50:31.767 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T21:50:31.788 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T21:50:31.788 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T21:50:31.788 DEBUG:teuthology.orchestra.run.trial163:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T21:50:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T21:50:31.792 DEBUG:teuthology.orchestra.run.trial163:> uname -r 2026-02-25T21:50:31.836 INFO:teuthology.orchestra.run.trial163.stdout:5.15.0-170-generic 2026-02-25T21:50:31.837 INFO:teuthology.task.kernel:Running kernel on trial163: 5.15.0-170-generic 2026-02-25T21:50:31.837 DEBUG:teuthology.orchestra.run.trial163:> sudo apt-get clean 2026-02-25T21:50:31.895 DEBUG:teuthology.orchestra.run.trial163:> sudo apt-get update 2026-02-25T21:50:32.410 INFO:teuthology.orchestra.run.trial163.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T21:50:32.501 INFO:teuthology.orchestra.run.trial163.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-25T21:50:32.840 INFO:teuthology.orchestra.run.trial163.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-25T21:50:32.990 INFO:teuthology.orchestra.run.trial163.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-25T21:50:33.130 INFO:teuthology.orchestra.run.trial163.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3274 kB] 2026-02-25T21:50:33.544 INFO:teuthology.orchestra.run.trial163.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [498 kB] 2026-02-25T21:50:33.554 INFO:teuthology.orchestra.run.trial163.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-25T21:50:33.554 INFO:teuthology.orchestra.run.trial163.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-25T21:50:33.598 INFO:teuthology.orchestra.run.trial163.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-25T21:50:33.607 INFO:teuthology.orchestra.run.trial163.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-25T21:50:33.607 INFO:teuthology.orchestra.run.trial163.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5400 kB] 2026-02-25T21:50:33.765 INFO:teuthology.orchestra.run.trial163.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1025 kB] 2026-02-25T21:50:33.803 INFO:teuthology.orchestra.run.trial163.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3007 kB] 2026-02-25T21:50:33.893 INFO:teuthology.orchestra.run.trial163.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [430 kB] 2026-02-25T21:50:33.911 INFO:teuthology.orchestra.run.trial163.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-25T21:50:33.911 INFO:teuthology.orchestra.run.trial163.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1019 kB] 2026-02-25T21:50:33.952 INFO:teuthology.orchestra.run.trial163.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB] 2026-02-25T21:50:33.959 INFO:teuthology.orchestra.run.trial163.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.6 kB] 2026-02-25T21:50:33.959 INFO:teuthology.orchestra.run.trial163.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5217 kB] 2026-02-25T21:50:34.108 INFO:teuthology.orchestra.run.trial163.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [993 kB] 2026-02-25T21:50:37.514 INFO:teuthology.orchestra.run.trial163.stdout:Fetched 23.1 MB in 3s (7829 kB/s) 2026-02-25T21:50:38.037 INFO:teuthology.orchestra.run.trial163.stdout:Reading package lists... 2026-02-25T21:50:38.051 DEBUG:teuthology.orchestra.run.trial163:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T21:50:38.084 INFO:teuthology.orchestra.run.trial163.stdout:Reading package lists... 2026-02-25T21:50:38.191 INFO:teuthology.orchestra.run.trial163.stdout:Building dependency tree... 2026-02-25T21:50:38.191 INFO:teuthology.orchestra.run.trial163.stdout:Reading state information... 2026-02-25T21:50:38.275 INFO:teuthology.orchestra.run.trial163.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: sg3-utils-udev 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout:The following additional packages will be installed: 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: linux-generic linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: linux-headers-generic linux-image-5.15.0-171-generic 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: linux-modules-5.15.0-171-generic linux-modules-extra-5.15.0-171-generic 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout:Suggested packages: 2026-02-25T21:50:38.276 INFO:teuthology.orchestra.run.trial163.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-25T21:50:38.288 INFO:teuthology.orchestra.run.trial163.stdout:The following NEW packages will be installed: 2026-02-25T21:50:38.288 INFO:teuthology.orchestra.run.trial163.stdout: linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T21:50:38.288 INFO:teuthology.orchestra.run.trial163.stdout: linux-image-5.15.0-171-generic linux-modules-5.15.0-171-generic 2026-02-25T21:50:38.289 INFO:teuthology.orchestra.run.trial163.stdout: linux-modules-extra-5.15.0-171-generic 2026-02-25T21:50:38.289 INFO:teuthology.orchestra.run.trial163.stdout:The following packages will be upgraded: 2026-02-25T21:50:38.289 INFO:teuthology.orchestra.run.trial163.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-25T21:50:43.883 INFO:teuthology.orchestra.run.trial163.stdout:3 upgraded, 5 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T21:50:43.883 INFO:teuthology.orchestra.run.trial163.stdout:Need to get 113 MB of archives. 2026-02-25T21:50:43.883 INFO:teuthology.orchestra.run.trial163.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-25T21:50:43.883 INFO:teuthology.orchestra.run.trial163.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-171-generic amd64 5.15.0-171.181 [22.7 MB] 2026-02-25T21:50:44.976 INFO:teuthology.orchestra.run.trial163.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-171-generic amd64 5.15.0-171.181 [11.6 MB] 2026-02-25T21:50:45.245 INFO:teuthology.orchestra.run.trial163.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-171-generic amd64 5.15.0-171.181 [63.9 MB] 2026-02-25T21:50:47.154 INFO:teuthology.orchestra.run.trial163.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.171.160 [1658 B] 2026-02-25T21:50:47.246 INFO:teuthology.orchestra.run.trial163.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.171.160 [2486 B] 2026-02-25T21:50:47.349 INFO:teuthology.orchestra.run.trial163.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171 all 5.15.0-171.181 [12.4 MB] 2026-02-25T21:50:48.381 INFO:teuthology.orchestra.run.trial163.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171-generic amd64 5.15.0-171.181 [2849 kB] 2026-02-25T21:50:48.502 INFO:teuthology.orchestra.run.trial163.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.171.160 [2332 B] 2026-02-25T21:50:48.702 INFO:teuthology.orchestra.run.trial163.stdout:Fetched 113 MB in 10s (11.1 MB/s) 2026-02-25T21:50:48.723 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package linux-modules-5.15.0-171-generic. 2026-02-25T21:50:48.892 INFO:teuthology.orchestra.run.trial163.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-25T21:50:48.894 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../0-linux-modules-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T21:50:48.896 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:49.224 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package linux-image-5.15.0-171-generic. 2026-02-25T21:50:49.232 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../1-linux-image-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T21:50:49.254 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:49.294 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-171-generic. 2026-02-25T21:50:49.302 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T21:50:49.303 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:50.403 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../3-linux-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T21:50:50.404 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T21:50:50.429 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T21:50:50.430 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-image-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T21:50:50.447 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package linux-headers-5.15.0-171. 2026-02-25T21:50:50.455 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../5-linux-headers-5.15.0-171_5.15.0-171.181_all.deb ... 2026-02-25T21:50:50.456 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T21:50:51.824 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package linux-headers-5.15.0-171-generic. 2026-02-25T21:50:51.838 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../6-linux-headers-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T21:50:51.839 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:52.430 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T21:50:52.431 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking linux-headers-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T21:50:52.457 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T21:50:52.458 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:52.459 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-headers-generic (5.15.0.171.160) ... 2026-02-25T21:50:52.461 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:53.256 INFO:teuthology.orchestra.run.trial163.stdout:I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-25T21:50:53.257 INFO:teuthology.orchestra.run.trial163.stdout:I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-170-generic 2026-02-25T21:50:53.257 INFO:teuthology.orchestra.run.trial163.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-171-generic 2026-02-25T21:50:53.257 INFO:teuthology.orchestra.run.trial163.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-171-generic 2026-02-25T21:50:53.264 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:54.084 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:54.896 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-image-generic (5.15.0.171.160) ... 2026-02-25T21:50:54.898 INFO:teuthology.orchestra.run.trial163.stdout:Setting up linux-generic (5.15.0.171.160) ... 2026-02-25T21:50:54.899 INFO:teuthology.orchestra.run.trial163.stdout:Processing triggers for linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T21:50:54.904 INFO:teuthology.orchestra.run.trial163.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-25T21:50:54.904 INFO:teuthology.orchestra.run.trial163.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-171-generic 2026-02-25T21:51:01.976 INFO:teuthology.orchestra.run.trial163.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-25T21:51:08.517 INFO:teuthology.orchestra.run.trial163.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-25T21:51:08.518 INFO:teuthology.orchestra.run.trial163.stdout:Sourcing file `/etc/default/grub' 2026-02-25T21:51:08.537 INFO:teuthology.orchestra.run.trial163.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T21:51:08.538 INFO:teuthology.orchestra.run.trial163.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T21:51:08.541 INFO:teuthology.orchestra.run.trial163.stdout:Generating grub configuration file ... 2026-02-25T21:51:08.624 INFO:teuthology.orchestra.run.trial163.stdout:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T21:51:08.630 INFO:teuthology.orchestra.run.trial163.stdout:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T21:51:08.809 INFO:teuthology.orchestra.run.trial163.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T21:51:08.815 INFO:teuthology.orchestra.run.trial163.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T21:51:08.861 INFO:teuthology.orchestra.run.trial163.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T21:51:08.866 INFO:teuthology.orchestra.run.trial163.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T21:51:08.941 INFO:teuthology.orchestra.run.trial163.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T21:51:08.941 INFO:teuthology.orchestra.run.trial163.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T21:51:08.941 INFO:teuthology.orchestra.run.trial163.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T21:51:08.943 INFO:teuthology.orchestra.run.trial163.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T21:51:08.954 INFO:teuthology.orchestra.run.trial163.stdout:done 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:Pending kernel upgrade! 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:Running kernel version: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 5.15.0-170-generic 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:Diagnostics: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: The currently running kernel version is not the expected kernel version 5.15.0-171-generic. 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:The processor microcode seems to be up-to-date. 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:No services need to be restarted. 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:No containers need to be restarted. 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout:No user sessions are running outdated binaries. 2026-02-25T21:51:09.361 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:51:09.362 INFO:teuthology.orchestra.run.trial163.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T21:51:10.193 DEBUG:teuthology.orchestra.run.trial163:> dpkg -s linux-image-generic 2026-02-25T21:51:10.203 INFO:teuthology.orchestra.run.trial163.stdout:Package: linux-image-generic 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Status: install ok installed 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Priority: optional 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Section: kernel 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Installed-Size: 22 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Architecture: amd64 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Source: linux-meta 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Version: 5.15.0.171.160 2026-02-25T21:51:10.204 INFO:teuthology.orchestra.run.trial163.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T21:51:10.205 INFO:teuthology.orchestra.run.trial163.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T21:51:10.205 INFO:teuthology.orchestra.run.trial163.stdout:Recommends: thermald 2026-02-25T21:51:10.205 INFO:teuthology.orchestra.run.trial163.stdout:Description: Generic Linux kernel image 2026-02-25T21:51:10.205 INFO:teuthology.orchestra.run.trial163.stdout: This package will always depend on the latest generic kernel image 2026-02-25T21:51:10.205 INFO:teuthology.orchestra.run.trial163.stdout: available. 2026-02-25T21:51:10.205 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-170-generic Expected: 5.15.0-171-generic 2026-02-25T21:51:10.206 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T21:51:10.206 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T21:51:10.206 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T21:51:10.206 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial163.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T21:51:10.206 DEBUG:teuthology.orchestra.run.trial163:> sudo apt-get clean 2026-02-25T21:51:10.285 DEBUG:teuthology.orchestra.run.trial163:> sudo apt-get update 2026-02-25T21:51:10.468 INFO:teuthology.orchestra.run.trial163.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T21:51:10.474 INFO:teuthology.orchestra.run.trial163.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T21:51:10.484 INFO:teuthology.orchestra.run.trial163.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T21:51:10.495 INFO:teuthology.orchestra.run.trial163.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T21:51:11.303 INFO:teuthology.orchestra.run.trial163.stdout:Reading package lists... 2026-02-25T21:51:11.317 DEBUG:teuthology.orchestra.run.trial163:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T21:51:11.349 INFO:teuthology.orchestra.run.trial163.stdout:Reading package lists... 2026-02-25T21:51:11.459 INFO:teuthology.orchestra.run.trial163.stdout:Building dependency tree... 2026-02-25T21:51:11.459 INFO:teuthology.orchestra.run.trial163.stdout:Reading state information... 2026-02-25T21:51:11.545 INFO:teuthology.orchestra.run.trial163.stdout:linux-image-generic is already the newest version (5.15.0.171.160). 2026-02-25T21:51:11.545 INFO:teuthology.orchestra.run.trial163.stdout:linux-image-generic set to manually installed. 2026-02-25T21:51:11.545 INFO:teuthology.orchestra.run.trial163.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T21:51:11.546 INFO:teuthology.orchestra.run.trial163.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T21:51:11.546 INFO:teuthology.orchestra.run.trial163.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T21:51:11.546 INFO:teuthology.orchestra.run.trial163.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T21:51:11.546 INFO:teuthology.orchestra.run.trial163.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T21:51:11.546 INFO:teuthology.orchestra.run.trial163.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T21:51:11.546 INFO:teuthology.orchestra.run.trial163.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T21:51:11.565 INFO:teuthology.orchestra.run.trial163.stdout:0 upgraded, 0 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T21:51:11.567 DEBUG:teuthology.orchestra.run.trial163:> dpkg -s linux-image-generic 2026-02-25T21:51:11.617 INFO:teuthology.orchestra.run.trial163.stdout:Package: linux-image-generic 2026-02-25T21:51:11.617 INFO:teuthology.orchestra.run.trial163.stdout:Status: install ok installed 2026-02-25T21:51:11.617 INFO:teuthology.orchestra.run.trial163.stdout:Priority: optional 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Section: kernel 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Installed-Size: 22 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Architecture: amd64 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Source: linux-meta 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Version: 5.15.0.171.160 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Recommends: thermald 2026-02-25T21:51:11.618 INFO:teuthology.orchestra.run.trial163.stdout:Description: Generic Linux kernel image 2026-02-25T21:51:11.619 INFO:teuthology.orchestra.run.trial163.stdout: This package will always depend on the latest generic kernel image 2026-02-25T21:51:11.619 INFO:teuthology.orchestra.run.trial163.stdout: available. 2026-02-25T21:51:11.620 DEBUG:teuthology.orchestra.run.trial163:> mktemp 2026-02-25T21:51:11.665 INFO:teuthology.orchestra.run.trial163.stdout:/tmp/tmp.4n7j1tHPzl 2026-02-25T21:51:11.665 DEBUG:teuthology.orchestra.run.trial163:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4n7j1tHPzl 2026-02-25T21:51:11.713 DEBUG:teuthology.orchestra.run.trial163:> sudo chmod 0666 /tmp/tmp.4n7j1tHPzl 2026-02-25T21:51:11.811 DEBUG:teuthology.orchestra.remote:trial163:/tmp/tmp.4n7j1tHPzl is 9KB 2026-02-25T21:51:11.861 DEBUG:teuthology.orchestra.run.trial163:> rm -fr /tmp/tmp.4n7j1tHPzl 2026-02-25T21:51:11.866 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-25T21:51:11.917 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:51:11.917 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-25T21:51:11.917 DEBUG:teuthology.orchestra.run.trial163:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-25T21:51:11.970 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-171-generic 2026-02-25T21:51:11.971 DEBUG:teuthology.orchestra.run.trial163:> sudo update-grub 2026-02-25T21:51:12.202 INFO:teuthology.orchestra.run.trial163.stderr:Sourcing file `/etc/default/grub' 2026-02-25T21:51:12.222 INFO:teuthology.orchestra.run.trial163.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T21:51:12.223 INFO:teuthology.orchestra.run.trial163.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T21:51:12.225 INFO:teuthology.orchestra.run.trial163.stderr:Generating grub configuration file ... 2026-02-25T21:51:12.308 INFO:teuthology.orchestra.run.trial163.stderr:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T21:51:12.314 INFO:teuthology.orchestra.run.trial163.stderr:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T21:51:12.488 INFO:teuthology.orchestra.run.trial163.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T21:51:12.494 INFO:teuthology.orchestra.run.trial163.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T21:51:12.536 INFO:teuthology.orchestra.run.trial163.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T21:51:12.542 INFO:teuthology.orchestra.run.trial163.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T21:51:12.613 INFO:teuthology.orchestra.run.trial163.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T21:51:12.613 INFO:teuthology.orchestra.run.trial163.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T21:51:12.613 INFO:teuthology.orchestra.run.trial163.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T21:51:12.615 INFO:teuthology.orchestra.run.trial163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T21:51:12.623 INFO:teuthology.orchestra.run.trial163.stderr:done 2026-02-25T21:51:12.625 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial163 2026-02-25T21:51:12.625 DEBUG:teuthology.orchestra.run.trial163:> sudo bash -c ' 2026-02-25T21:51:12.626 DEBUG:teuthology.orchestra.run.trial163:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T21:51:12.626 DEBUG:teuthology.orchestra.run.trial163:> ' 2026-02-25T21:51:42.629 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T21:51:42.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial163.front.sepia.ceph.com' 2026-02-25T21:51:42.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T21:52:42.692 DEBUG:teuthology.orchestra.remote:timed out 2026-02-25T21:52:51.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial163.front.sepia.ceph.com' 2026-02-25T21:52:51.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T21:52:51.829 DEBUG:teuthology.orchestra.run.trial163:> true 2026-02-25T21:52:52.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial163.front.sepia.ceph.com' 2026-02-25T21:52:52.228 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-171-generic"... 2026-02-25T21:52:52.228 DEBUG:teuthology.orchestra.run.trial163:> uname -r 2026-02-25T21:52:52.279 INFO:teuthology.orchestra.run.trial163.stdout:5.15.0-171-generic 2026-02-25T21:52:52.280 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-171-generic vs 5.15.0-171-generic 2026-02-25T21:52:52.280 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T21:52:52.280 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-25T21:52:53.281 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T21:52:53.281 DEBUG:teuthology.orchestra.run.trial163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T21:52:53.294 INFO:teuthology.orchestra.run.trial163.stdout:ttyS1 2026-02-25T21:52:53.322 DEBUG:teuthology.parallel:result is None 2026-02-25T21:52:53.322 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T21:52:53.334 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T21:52:53.334 DEBUG:teuthology.orchestra.run.trial163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T21:52:53.367 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T21:52:53.379 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T21:52:53.391 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T21:52:53.392 DEBUG:teuthology.orchestra.run.trial163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T21:52:53.416 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T21:52:53.428 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T21:52:53.429 DEBUG:teuthology.orchestra.run.trial163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T21:52:53.464 INFO:teuthology.orchestra.run.trial163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:52:53.470 INFO:teuthology.orchestra.run.trial163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:52:53.472 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T21:52:53.483 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T21:52:53.484 DEBUG:teuthology.orchestra.run.trial163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T21:52:53.524 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T21:52:53.540 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T21:52:53.540 DEBUG:teuthology.orchestra.run.trial163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T21:52:53.571 DEBUG:teuthology.orchestra.run.trial163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T21:52:53.615 DEBUG:teuthology.orchestra.run.trial163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T21:52:53.660 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:52:53.660 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T21:52:53.716 DEBUG:teuthology.orchestra.run.trial163:> sudo service rsyslog restart 2026-02-25T21:52:53.777 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T21:52:53.789 INFO:teuthology.task.internal:Starting timer... 2026-02-25T21:52:53.789 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T21:52:53.812 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T21:52:53.829 DEBUG:teuthology.task.selinux:Excluding trial163: OS 'ubuntu' does not support SELinux 2026-02-25T21:52:53.829 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T21:52:53.830 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-25T21:52:53.830 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T21:52:53.830 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T21:52:53.845 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T21:52:53.857 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T21:52:53.858 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T21:55:14.464 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial163.front.sepia.ceph.com')] 2026-02-25T21:55:14.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial163.front.sepia.ceph.com' 2026-02-25T21:55:14.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T21:55:14.569 DEBUG:teuthology.orchestra.run.trial163:> true 2026-02-25T21:55:14.820 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial163.front.sepia.ceph.com' 2026-02-25T21:55:14.820 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T21:55:14.833 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T21:55:14.833 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T21:55:14.834 DEBUG:teuthology.orchestra.run.trial163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Command line: ntpd -gq 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: ---------------------------------------------------- 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: ntp-4 is maintained by Network Time Foundation, 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: corporation. Support and training for ntp-4 are 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: available at https://www.nwtime.org/support 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: ---------------------------------------------------- 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: proto: precision = 0.030 usec (-25) 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: basedate set to 2022-02-04 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: gps base set to 2022-02-06 (week 2196) 2026-02-25T21:55:14.881 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: restrict ::: KOD does nothing without LIMITED. 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listen and drop on 0 v6wildcard [::]:123 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listen normally on 2 lo 127.0.0.1:123 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listen normally on 3 enp1s0f0 10.20.193.163:123 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listen normally on 4 lo [::1]:123 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6374%2]:123 2026-02-25T21:55:14.882 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:15 ntpd[8806]: Listening on routing socket on fd #22 for interface updates 2026-02-25T21:55:15.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:15.882 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:15.882 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:15.882 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:15.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:15.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:15.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:15.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:16 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:17.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:18 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:17.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:18 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:17.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:18 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:17.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:18 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:17.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:18 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:17.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:18 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:18.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:19 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:18.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:19 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:18.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:19 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:18.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:19 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:18.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:19 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:18.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:19 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:19.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:19.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:19.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:19.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:19.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:19.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:19.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:19.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:19.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:19.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:19.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:19.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:20 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:20.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:20.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:20.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:20.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:20.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:20.946 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:21.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:21.881 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:21.882 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:21.882 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:21.899 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:21.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:21.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:21.900 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:22 ntpd[8806]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stdout:25 Feb 21:55:21 ntpd[8806]: ntpd: time set -0.465224 s 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stdout:ntpd: time set -0.465224s 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-25T21:55:21.953 INFO:teuthology.orchestra.run.trial163.stderr:25 Feb 21:55:21 ntpd[8806]: can't open /var/log/ntpstats/loopstats.20260225: Permission denied 2026-02-25T21:55:21.977 INFO:teuthology.orchestra.run.trial163.stdout: remote refid st t when poll reach delay offset jitter 2026-02-25T21:55:21.977 INFO:teuthology.orchestra.run.trial163.stdout:============================================================================== 2026-02-25T21:55:21.977 INFO:teuthology.orchestra.run.trial163.stdout: 108.61.215.221 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T21:55:21.977 INFO:teuthology.orchestra.run.trial163.stdout: 149.248.12.167 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T21:55:21.977 INFO:teuthology.orchestra.run.trial163.stdout: lax1.us.ntp.li .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T21:55:21.977 INFO:teuthology.orchestra.run.trial163.stdout: 199.101.96.52 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-25T21:55:21.978 INFO:teuthology.run_tasks:Running task install... 2026-02-25T21:55:21.992 DEBUG:teuthology.task.install:project ceph 2026-02-25T21:55:21.992 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'}} 2026-02-25T21:55:21.992 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'} 2026-02-25T21:55:21.993 INFO:teuthology.task.install:Using flavor: default 2026-02-25T21:55:21.998 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T21:55:21.998 INFO:teuthology.task.install:extra packages: [] 2026-02-25T21:55:21.999 DEBUG:teuthology.orchestra.run.trial163:> sudo apt-key list | grep Ceph 2026-02-25T21:55:22.066 INFO:teuthology.orchestra.run.trial163.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-25T21:55:22.095 INFO:teuthology.orchestra.run.trial163.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-25T21:55:22.095 INFO:teuthology.orchestra.run.trial163.stdout:uid [ unknown] Ceph.com (release key) 2026-02-25T21:55:22.096 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-osd, ceph-osd-classic, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-25T21:55:22.096 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T21:55:22.221 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default/ 2026-02-25T21:55:22.221 INFO:teuthology.task.install.deb:Package version is 20.3.0-5305-g0886691e-1jammy 2026-02-25T21:55:22.395 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:55:22.395 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-25T21:55:22.405 DEBUG:teuthology.orchestra.run.trial163:> sudo apt-get update 2026-02-25T21:55:22.580 INFO:teuthology.orchestra.run.trial163.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy InRelease 2026-02-25T21:55:22.591 INFO:teuthology.orchestra.run.trial163.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy Release [7689 B] 2026-02-25T21:55:22.603 INFO:teuthology.orchestra.run.trial163.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-25T21:55:22.615 INFO:teuthology.orchestra.run.trial163.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [19.2 kB] 2026-02-25T21:55:22.836 INFO:teuthology.orchestra.run.trial163.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T21:55:22.902 INFO:teuthology.orchestra.run.trial163.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T21:55:22.974 INFO:teuthology.orchestra.run.trial163.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T21:55:23.046 INFO:teuthology.orchestra.run.trial163.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T21:55:23.187 INFO:teuthology.orchestra.run.trial163.stdout:Fetched 26.9 kB in 1s (43.4 kB/s) 2026-02-25T21:55:23.706 INFO:teuthology.orchestra.run.trial163.stdout:Reading package lists... 2026-02-25T21:55:23.719 DEBUG:teuthology.orchestra.run.trial163:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.3.0-5305-g0886691e-1jammy cephadm=20.3.0-5305-g0886691e-1jammy ceph-mds=20.3.0-5305-g0886691e-1jammy ceph-mgr=20.3.0-5305-g0886691e-1jammy ceph-osd=20.3.0-5305-g0886691e-1jammy ceph-osd-classic=20.3.0-5305-g0886691e-1jammy ceph-common=20.3.0-5305-g0886691e-1jammy ceph-fuse=20.3.0-5305-g0886691e-1jammy ceph-test=20.3.0-5305-g0886691e-1jammy ceph-volume=20.3.0-5305-g0886691e-1jammy radosgw=20.3.0-5305-g0886691e-1jammy python3-rados=20.3.0-5305-g0886691e-1jammy python3-rgw=20.3.0-5305-g0886691e-1jammy python3-cephfs=20.3.0-5305-g0886691e-1jammy python3-rbd=20.3.0-5305-g0886691e-1jammy libcephfs2=20.3.0-5305-g0886691e-1jammy libcephfs-dev=20.3.0-5305-g0886691e-1jammy librados2=20.3.0-5305-g0886691e-1jammy librbd1=20.3.0-5305-g0886691e-1jammy rbd-fuse=20.3.0-5305-g0886691e-1jammy 2026-02-25T21:55:23.752 INFO:teuthology.orchestra.run.trial163.stdout:Reading package lists... 2026-02-25T21:55:23.862 INFO:teuthology.orchestra.run.trial163.stdout:Building dependency tree... 2026-02-25T21:55:23.862 INFO:teuthology.orchestra.run.trial163.stdout:Reading state information... 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T21:55:23.951 INFO:teuthology.orchestra.run.trial163.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout:The following additional packages will be installed: 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon jq libcephfs-daemon 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: nvme-cli python-asyncssh-doc python3-asyncssh python3-cachetools 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-ceph-argparse python3-ceph-common python3-cheroot python3-cherrypy3 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-google-auth python3-iniconfig python3-jaraco.classes 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-joblib python3-kubernetes python3-natsort python3-pluggy 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-pygments python3-pytest python3-repoze.lru python3-requests-oauthlib 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-routes python3-rsa python3-simplejson python3-sklearn 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-wcwidth python3-webob python3-websocket python3-zc.lockfile 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: qttranslations5-l10n socat xmlstarlet 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout:Suggested packages: 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout:Recommended packages: 2026-02-25T21:55:23.952 INFO:teuthology.orchestra.run.trial163.stdout: btrfs-tools 2026-02-25T21:55:23.987 INFO:teuthology.orchestra.run.trial163.stdout:The following NEW packages will be installed: 2026-02-25T21:55:23.987 INFO:teuthology.orchestra.run.trial163.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-25T21:55:23.987 INFO:teuthology.orchestra.run.trial163.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-25T21:55:23.987 INFO:teuthology.orchestra.run.trial163.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-osd-classic ceph-test 2026-02-25T21:55:23.987 INFO:teuthology.orchestra.run.trial163.stdout: ceph-volume cephadm jq libcephfs-daemon libcephfs-dev libcephfs-proxy2 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-25T21:55:23.988 INFO:teuthology.orchestra.run.trial163.stdout: socat xmlstarlet 2026-02-25T21:55:24.068 INFO:teuthology.orchestra.run.trial163.stdout:0 upgraded, 87 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T21:55:24.068 INFO:teuthology.orchestra.run.trial163.stdout:Need to get 702 MB of archives. 2026-02-25T21:55:24.068 INFO:teuthology.orchestra.run.trial163.stdout:After this operation, 2670 MB of additional disk space will be used. 2026-02-25T21:55:24.068 INFO:teuthology.orchestra.run.trial163.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.3.0-5305-g0886691e-1jammy [3797 kB] 2026-02-25T21:55:24.407 INFO:teuthology.orchestra.run.trial163.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-25T21:55:24.600 INFO:teuthology.orchestra.run.trial163.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.3.0-5305-g0886691e-1jammy [2910 kB] 2026-02-25T21:55:24.819 INFO:teuthology.orchestra.run.trial163.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-25T21:55:24.832 INFO:teuthology.orchestra.run.trial163.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-25T21:55:24.907 INFO:teuthology.orchestra.run.trial163.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.3.0-5305-g0886691e-1jammy [882 kB] 2026-02-25T21:55:24.916 INFO:teuthology.orchestra.run.trial163.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-25T21:55:24.929 INFO:teuthology.orchestra.run.trial163.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.3.0-5305-g0886691e-1jammy [359 kB] 2026-02-25T21:55:24.940 INFO:teuthology.orchestra.run.trial163.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.3.0-5305-g0886691e-1jammy [32.8 kB] 2026-02-25T21:55:24.940 INFO:teuthology.orchestra.run.trial163.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.3.0-5305-g0886691e-1jammy [225 kB] 2026-02-25T21:55:24.945 INFO:teuthology.orchestra.run.trial163.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.3.0-5305-g0886691e-1jammy [88.4 kB] 2026-02-25T21:55:24.946 INFO:teuthology.orchestra.run.trial163.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.3.0-5305-g0886691e-1jammy [338 kB] 2026-02-25T21:55:24.955 INFO:teuthology.orchestra.run.trial163.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.3.0-5305-g0886691e-1jammy [9092 kB] 2026-02-25T21:55:25.160 INFO:teuthology.orchestra.run.trial163.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-25T21:55:25.173 INFO:teuthology.orchestra.run.trial163.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-25T21:55:25.205 INFO:teuthology.orchestra.run.trial163.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-25T21:55:25.215 INFO:teuthology.orchestra.run.trial163.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-25T21:55:25.217 INFO:teuthology.orchestra.run.trial163.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-25T21:55:25.219 INFO:teuthology.orchestra.run.trial163.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-25T21:55:25.220 INFO:teuthology.orchestra.run.trial163.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-25T21:55:25.240 INFO:teuthology.orchestra.run.trial163.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-25T21:55:25.240 INFO:teuthology.orchestra.run.trial163.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-25T21:55:25.248 INFO:teuthology.orchestra.run.trial163.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-25T21:55:25.329 INFO:teuthology.orchestra.run.trial163.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-25T21:55:25.329 INFO:teuthology.orchestra.run.trial163.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-25T21:55:25.329 INFO:teuthology.orchestra.run.trial163.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-25T21:55:25.330 INFO:teuthology.orchestra.run.trial163.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-25T21:55:25.330 INFO:teuthology.orchestra.run.trial163.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-25T21:55:25.330 INFO:teuthology.orchestra.run.trial163.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-25T21:55:25.330 INFO:teuthology.orchestra.run.trial163.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-25T21:55:25.336 INFO:teuthology.orchestra.run.trial163.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-25T21:55:25.336 INFO:teuthology.orchestra.run.trial163.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-25T21:55:25.420 INFO:teuthology.orchestra.run.trial163.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-25T21:55:25.420 INFO:teuthology.orchestra.run.trial163.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.3.0-5305-g0886691e-1jammy [113 kB] 2026-02-25T21:55:25.421 INFO:teuthology.orchestra.run.trial163.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.3.0-5305-g0886691e-1jammy [261 kB] 2026-02-25T21:55:25.423 INFO:teuthology.orchestra.run.trial163.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-25T21:55:25.423 INFO:teuthology.orchestra.run.trial163.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.3.0-5305-g0886691e-1jammy [31.5 MB] 2026-02-25T21:55:25.423 INFO:teuthology.orchestra.run.trial163.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-25T21:55:25.424 INFO:teuthology.orchestra.run.trial163.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-25T21:55:25.454 INFO:teuthology.orchestra.run.trial163.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-25T21:55:25.457 INFO:teuthology.orchestra.run.trial163.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-25T21:55:25.457 INFO:teuthology.orchestra.run.trial163.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-25T21:55:25.527 INFO:teuthology.orchestra.run.trial163.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-25T21:55:25.527 INFO:teuthology.orchestra.run.trial163.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-25T21:55:25.527 INFO:teuthology.orchestra.run.trial163.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-25T21:55:25.528 INFO:teuthology.orchestra.run.trial163.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-25T21:55:25.528 INFO:teuthology.orchestra.run.trial163.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-25T21:55:25.529 INFO:teuthology.orchestra.run.trial163.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-25T21:55:25.602 INFO:teuthology.orchestra.run.trial163.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-25T21:55:25.603 INFO:teuthology.orchestra.run.trial163.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-25T21:55:25.605 INFO:teuthology.orchestra.run.trial163.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-25T21:55:25.690 INFO:teuthology.orchestra.run.trial163.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-25T21:55:25.693 INFO:teuthology.orchestra.run.trial163.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-25T21:55:25.696 INFO:teuthology.orchestra.run.trial163.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-25T21:55:25.700 INFO:teuthology.orchestra.run.trial163.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-25T21:55:25.703 INFO:teuthology.orchestra.run.trial163.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-25T21:55:25.703 INFO:teuthology.orchestra.run.trial163.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-25T21:55:25.704 INFO:teuthology.orchestra.run.trial163.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-25T21:55:25.778 INFO:teuthology.orchestra.run.trial163.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-25T21:55:25.778 INFO:teuthology.orchestra.run.trial163.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-25T21:55:25.786 INFO:teuthology.orchestra.run.trial163.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-25T21:55:25.866 INFO:teuthology.orchestra.run.trial163.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-25T21:55:25.868 INFO:teuthology.orchestra.run.trial163.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-25T21:55:25.868 INFO:teuthology.orchestra.run.trial163.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-25T21:55:25.869 INFO:teuthology.orchestra.run.trial163.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-25T21:55:27.442 INFO:teuthology.orchestra.run.trial163.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.3.0-5305-g0886691e-1jammy [4983 kB] 2026-02-25T21:55:27.869 INFO:teuthology.orchestra.run.trial163.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.3.0-5305-g0886691e-1jammy [293 kB] 2026-02-25T21:55:27.914 INFO:teuthology.orchestra.run.trial163.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.3.0-5305-g0886691e-1jammy [127 kB] 2026-02-25T21:55:27.919 INFO:teuthology.orchestra.run.trial163.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.3.0-5305-g0886691e-1jammy [922 kB] 2026-02-25T21:55:28.008 INFO:teuthology.orchestra.run.trial163.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.3.0-5305-g0886691e-1jammy [7541 kB] 2026-02-25T21:55:28.659 INFO:teuthology.orchestra.run.trial163.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-classic amd64 20.3.0-5305-g0886691e-1jammy [468 MB] 2026-02-25T21:55:55.203 INFO:teuthology.orchestra.run.trial163.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.3.0-5305-g0886691e-1jammy [180 kB] 2026-02-25T21:55:55.203 INFO:teuthology.orchestra.run.trial163.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.3.0-5305-g0886691e-1jammy [14.0 kB] 2026-02-25T21:55:55.204 INFO:teuthology.orchestra.run.trial163.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.3.0-5305-g0886691e-1jammy [1030 kB] 2026-02-25T21:55:55.209 INFO:teuthology.orchestra.run.trial163.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.3.0-5305-g0886691e-1jammy [2437 kB] 2026-02-25T21:55:55.222 INFO:teuthology.orchestra.run.trial163.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.3.0-5305-g0886691e-1jammy [1069 kB] 2026-02-25T21:55:55.227 INFO:teuthology.orchestra.run.trial163.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.3.0-5305-g0886691e-1jammy [190 kB] 2026-02-25T21:55:55.228 INFO:teuthology.orchestra.run.trial163.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.3.0-5305-g0886691e-1jammy [54.7 MB] 2026-02-25T21:55:56.398 INFO:teuthology.orchestra.run.trial163.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.3.0-5305-g0886691e-1jammy [8626 kB] 2026-02-25T21:55:56.582 INFO:teuthology.orchestra.run.trial163.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.3.0-5305-g0886691e-1jammy [14.2 kB] 2026-02-25T21:55:56.582 INFO:teuthology.orchestra.run.trial163.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.3.0-5305-g0886691e-1jammy [68.6 MB] 2026-02-25T21:55:58.105 INFO:teuthology.orchestra.run.trial163.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.3.0-5305-g0886691e-1jammy [137 kB] 2026-02-25T21:55:58.105 INFO:teuthology.orchestra.run.trial163.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.3.0-5305-g0886691e-1jammy [45.8 kB] 2026-02-25T21:55:58.106 INFO:teuthology.orchestra.run.trial163.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.3.0-5305-g0886691e-1jammy [32.4 kB] 2026-02-25T21:55:58.106 INFO:teuthology.orchestra.run.trial163.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.3.0-5305-g0886691e-1jammy [44.2 kB] 2026-02-25T21:55:58.106 INFO:teuthology.orchestra.run.trial163.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.3.0-5305-g0886691e-1jammy [18.3 MB] 2026-02-25T21:55:58.507 INFO:teuthology.orchestra.run.trial163.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.3.0-5305-g0886691e-1jammy [94.1 kB] 2026-02-25T21:55:58.792 INFO:teuthology.orchestra.run.trial163.stdout:Fetched 702 MB in 35s (20.3 MB/s) 2026-02-25T21:55:58.808 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-25T21:55:58.976 INFO:teuthology.orchestra.run.trial163.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 202926 files and directories currently installed.) 2026-02-25T21:55:58.979 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-25T21:55:58.980 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T21:55:59.000 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-25T21:55:59.014 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-25T21:55:59.015 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T21:55:59.033 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-25T21:55:59.046 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-25T21:55:59.046 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T21:55:59.064 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-25T21:55:59.073 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T21:55:59.078 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T21:55:59.109 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-25T21:55:59.119 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T21:55:59.120 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T21:55:59.137 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-25T21:55:59.146 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T21:55:59.147 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T21:55:59.168 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-25T21:55:59.178 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-25T21:55:59.178 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T21:55:59.194 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package librados2. 2026-02-25T21:55:59.204 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../07-librados2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.204 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking librados2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.248 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libnbd0. 2026-02-25T21:55:59.258 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-25T21:55:59.259 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-25T21:55:59.273 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package librbd1. 2026-02-25T21:55:59.283 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../09-librbd1_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.283 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking librbd1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.322 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libcephfs2. 2026-02-25T21:55:59.332 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../10-libcephfs2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.332 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libcephfs2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.353 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-rados. 2026-02-25T21:55:59.362 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../11-python3-rados_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.363 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-rados (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.381 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-25T21:55:59.391 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:55:59.391 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-ceph-argparse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.406 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-cephfs. 2026-02-25T21:55:59.416 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../13-python3-cephfs_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.417 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-cephfs (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.433 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-25T21:55:59.443 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../14-python3-ceph-common_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:55:59.443 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.463 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-25T21:55:59.473 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-25T21:55:59.473 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T21:55:59.489 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-prettytable. 2026-02-25T21:55:59.499 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-25T21:55:59.499 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-25T21:55:59.514 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-rbd. 2026-02-25T21:55:59.524 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../17-python3-rbd_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.525 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-rbd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.542 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-25T21:55:59.552 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-25T21:55:59.552 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T21:55:59.571 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package librgw2. 2026-02-25T21:55:59.581 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../19-librgw2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.582 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking librgw2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.684 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-rgw. 2026-02-25T21:55:59.695 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../20-python3-rgw_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.695 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-rgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.711 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-25T21:55:59.721 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-25T21:55:59.721 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T21:55:59.736 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libradosstriper1. 2026-02-25T21:55:59.746 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../22-libradosstriper1_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.747 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libradosstriper1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:55:59.764 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-common. 2026-02-25T21:55:59.773 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../23-ceph-common_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:55:59.774 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:00.048 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-base. 2026-02-25T21:56:00.058 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../24-ceph-base_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:00.063 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-base (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:00.125 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-25T21:56:00.135 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-25T21:56:00.136 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-25T21:56:00.151 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-cheroot. 2026-02-25T21:56:00.161 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-25T21:56:00.161 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T21:56:00.179 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-25T21:56:00.189 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-25T21:56:00.189 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-25T21:56:00.205 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-25T21:56:00.215 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-25T21:56:00.215 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-25T21:56:00.231 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-25T21:56:00.241 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-25T21:56:00.241 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-25T21:56:00.257 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-tempora. 2026-02-25T21:56:00.267 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-25T21:56:00.267 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-25T21:56:00.283 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-portend. 2026-02-25T21:56:00.293 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-25T21:56:00.293 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-25T21:56:00.308 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-25T21:56:00.318 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-25T21:56:00.319 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-25T21:56:00.334 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-25T21:56:00.344 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-25T21:56:00.344 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-25T21:56:00.368 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-natsort. 2026-02-25T21:56:00.377 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-25T21:56:00.378 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-25T21:56:00.395 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-25T21:56:00.405 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:56:00.405 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mgr-modules-core (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:00.436 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-25T21:56:00.446 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:00.446 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libsqlite3-mod-ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:00.462 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mgr. 2026-02-25T21:56:00.472 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../37-ceph-mgr_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:00.472 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mgr (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:00.495 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mon. 2026-02-25T21:56:00.504 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../38-ceph-mon_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:00.504 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:00.571 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-25T21:56:00.581 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-25T21:56:00.581 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T21:56:00.598 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-osd-classic. 2026-02-25T21:56:00.608 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../40-ceph-osd-classic_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:00.608 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-osd-classic (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.636 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-osd. 2026-02-25T21:56:04.651 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../41-ceph-osd_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:04.652 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-osd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.673 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph. 2026-02-25T21:56:04.688 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../42-ceph_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:04.689 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.707 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-fuse. 2026-02-25T21:56:04.721 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../43-ceph-fuse_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:04.722 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.747 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mds. 2026-02-25T21:56:04.760 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../44-ceph-mds_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:04.760 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mds (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.794 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package cephadm. 2026-02-25T21:56:04.804 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../45-cephadm_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:04.805 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.823 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-25T21:56:04.833 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../46-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T21:56:04.833 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T21:56:04.855 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-25T21:56:04.865 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../47-ceph-mgr-cephadm_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:56:04.865 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mgr-cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:04.888 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-25T21:56:04.898 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2026-02-25T21:56:04.898 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-25T21:56:04.914 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-routes. 2026-02-25T21:56:04.924 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../49-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-25T21:56:04.924 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T21:56:04.946 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-25T21:56:04.956 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../50-ceph-mgr-dashboard_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:56:04.956 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mgr-dashboard (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:05.639 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-25T21:56:05.650 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../51-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-25T21:56:05.650 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T21:56:05.690 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-joblib. 2026-02-25T21:56:05.700 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../52-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-25T21:56:05.700 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T21:56:05.728 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-25T21:56:05.739 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../53-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-25T21:56:05.739 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-25T21:56:05.755 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-sklearn. 2026-02-25T21:56:05.765 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../54-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-25T21:56:05.765 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T21:56:05.848 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-25T21:56:05.859 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../55-ceph-mgr-diskprediction-local_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:56:05.859 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mgr-diskprediction-local (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:06.020 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-cachetools. 2026-02-25T21:56:06.030 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../56-python3-cachetools_5.0.0-1_all.deb ... 2026-02-25T21:56:06.030 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-25T21:56:06.045 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-rsa. 2026-02-25T21:56:06.056 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../57-python3-rsa_4.8-1_all.deb ... 2026-02-25T21:56:06.056 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-25T21:56:06.074 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-google-auth. 2026-02-25T21:56:06.085 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../58-python3-google-auth_1.5.1-3_all.deb ... 2026-02-25T21:56:06.085 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-25T21:56:06.103 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-25T21:56:06.113 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../59-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-25T21:56:06.113 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T21:56:06.130 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-websocket. 2026-02-25T21:56:06.141 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../60-python3-websocket_1.2.3-1_all.deb ... 2026-02-25T21:56:06.141 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-25T21:56:06.160 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-25T21:56:06.170 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../61-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-25T21:56:06.170 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T21:56:06.265 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-25T21:56:06.276 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../62-ceph-mgr-k8sevents_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:56:06.276 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-mgr-k8sevents (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:06.291 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-25T21:56:06.301 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../63-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-25T21:56:06.302 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T21:56:06.319 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-25T21:56:06.329 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../64-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T21:56:06.329 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T21:56:06.345 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package jq. 2026-02-25T21:56:06.355 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../65-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T21:56:06.355 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-25T21:56:06.371 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package socat. 2026-02-25T21:56:06.381 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../66-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-25T21:56:06.381 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-25T21:56:06.401 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package xmlstarlet. 2026-02-25T21:56:06.412 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../67-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-25T21:56:06.412 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-25T21:56:06.449 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-test. 2026-02-25T21:56:06.459 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../68-ceph-test_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:06.459 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-test (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.043 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package ceph-volume. 2026-02-25T21:56:07.054 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../69-ceph-volume_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T21:56:07.054 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking ceph-volume (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.079 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-25T21:56:07.089 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../70-libcephfs-daemon_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:07.089 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libcephfs-daemon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.105 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-25T21:56:07.116 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../71-libcephfs-proxy2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:07.116 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libcephfs-proxy2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.131 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-25T21:56:07.141 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../72-libcephfs-dev_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:07.141 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking libcephfs-dev (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.158 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package nvme-cli. 2026-02-25T21:56:07.169 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../73-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-25T21:56:07.169 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T21:56:07.201 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-25T21:56:07.211 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../74-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T21:56:07.212 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T21:56:07.242 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-25T21:56:07.253 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../75-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-25T21:56:07.253 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-25T21:56:07.269 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-pluggy. 2026-02-25T21:56:07.279 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../76-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-25T21:56:07.279 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-25T21:56:07.296 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-psutil. 2026-02-25T21:56:07.307 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../77-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-25T21:56:07.307 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-25T21:56:07.327 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-py. 2026-02-25T21:56:07.337 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../78-python3-py_1.10.0-1_all.deb ... 2026-02-25T21:56:07.337 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-25T21:56:07.359 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-pygments. 2026-02-25T21:56:07.369 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../79-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-25T21:56:07.369 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T21:56:07.413 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-toml. 2026-02-25T21:56:07.423 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../80-python3-toml_0.10.2-1_all.deb ... 2026-02-25T21:56:07.423 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-25T21:56:07.439 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-pytest. 2026-02-25T21:56:07.448 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../81-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-25T21:56:07.448 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T21:56:07.471 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-simplejson. 2026-02-25T21:56:07.481 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../82-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-25T21:56:07.481 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-25T21:56:07.507 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package python3-webob. 2026-02-25T21:56:07.518 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../83-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-25T21:56:07.518 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T21:56:07.535 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-25T21:56:07.546 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../84-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-25T21:56:07.546 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-25T21:56:07.614 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package radosgw. 2026-02-25T21:56:07.625 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../85-radosgw_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:07.625 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking radosgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.782 INFO:teuthology.orchestra.run.trial163.stdout:Selecting previously unselected package rbd-fuse. 2026-02-25T21:56:07.793 INFO:teuthology.orchestra.run.trial163.stdout:Preparing to unpack .../86-rbd-fuse_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T21:56:07.793 INFO:teuthology.orchestra.run.trial163.stdout:Unpacking rbd-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:07.822 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-25T21:56:07.887 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T21:56:07.887 INFO:teuthology.orchestra.run.trial163.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T21:56:07.960 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T21:56:08.236 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-25T21:56:08.607 INFO:teuthology.orchestra.run.trial163.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-25T21:56:08.618 INFO:teuthology.orchestra.run.trial163.stdout:Setting up cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:08.673 INFO:teuthology.orchestra.run.trial163.stdout:Adding system user cephadm....done 2026-02-25T21:56:08.679 INFO:teuthology.orchestra.run.trial163.stdout:usermod: no changes 2026-02-25T21:56:08.685 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-25T21:56:08.760 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T21:56:08.761 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-25T21:56:08.833 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-25T21:56:08.907 INFO:teuthology.orchestra.run.trial163.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T21:56:08.908 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-25T21:56:09.002 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T21:56:09.126 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-25T21:56:09.203 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-25T21:56:09.278 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-ceph-argparse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:09.346 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T21:56:09.347 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-25T21:56:09.348 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T21:56:09.349 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T21:56:09.350 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-25T21:56:09.468 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-25T21:56:09.545 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libcephfs-proxy2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:09.546 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T21:56:09.624 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-25T21:56:09.714 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T21:56:09.963 INFO:teuthology.orchestra.run.trial163.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-25T21:56:09.964 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T21:56:10.059 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T21:56:10.183 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T21:56:10.264 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-25T21:56:10.337 INFO:teuthology.orchestra.run.trial163.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-25T21:56:10.338 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:10.426 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T21:56:10.917 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T21:56:10.921 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-25T21:56:11.000 INFO:teuthology.orchestra.run.trial163.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T21:56:11.000 INFO:teuthology.orchestra.run.trial163.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-25T21:56:11.001 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-25T21:56:11.074 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-25T21:56:11.148 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T21:56:11.148 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-25T21:56:11.225 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-25T21:56:11.298 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-25T21:56:11.368 INFO:teuthology.orchestra.run.trial163.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T21:56:11.369 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-25T21:56:11.455 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T21:56:11.456 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T21:56:11.532 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T21:56:11.615 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-25T21:56:11.689 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T21:56:11.690 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T21:56:11.821 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-25T21:56:11.892 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T21:56:11.893 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-25T21:56:11.977 INFO:teuthology.orchestra.run.trial163.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-25T21:56:11.977 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-25T21:56:12.105 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T21:56:12.106 INFO:teuthology.orchestra.run.trial163.stdout:Setting up librados2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.107 INFO:teuthology.orchestra.run.trial163.stdout:Setting up librgw2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.108 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libsqlite3-mod-ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.109 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T21:56:12.602 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libcephfs2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.603 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libradosstriper1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.604 INFO:teuthology.orchestra.run.trial163.stdout:Setting up librbd1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.605 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mgr-modules-core (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.606 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:12.677 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T21:56:12.677 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T21:56:13.062 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libcephfs-dev (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.063 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-rados (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.064 INFO:teuthology.orchestra.run.trial163.stdout:Setting up libcephfs-daemon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.064 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-rbd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.065 INFO:teuthology.orchestra.run.trial163.stdout:Setting up rbd-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.066 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-rgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.067 INFO:teuthology.orchestra.run.trial163.stdout:Setting up python3-cephfs (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.068 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.096 INFO:teuthology.orchestra.run.trial163.stdout:Adding group ceph....done 2026-02-25T21:56:13.131 INFO:teuthology.orchestra.run.trial163.stdout:Adding system user ceph....done 2026-02-25T21:56:13.142 INFO:teuthology.orchestra.run.trial163.stdout:Setting system user ceph properties....done 2026-02-25T21:56:13.148 INFO:teuthology.orchestra.run.trial163.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-25T21:56:13.230 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-25T21:56:13.503 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-25T21:56:13.863 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-test (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:13.863 INFO:teuthology.orchestra.run.trial163.stdout:Setting up radosgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:14.117 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T21:56:14.118 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T21:56:14.477 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-base (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:14.588 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-25T21:56:14.944 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mds (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:15.005 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T21:56:15.005 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T21:56:15.370 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mgr (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:15.450 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T21:56:15.450 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T21:56:15.825 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mgr-k8sevents (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:15.826 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-osd-classic (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:15.843 INFO:teuthology.orchestra.run.trial163.stdout:update-alternatives: using /usr/bin/ceph-osd-classic to provide /usr/bin/ceph-osd (ceph-osd) in auto mode 2026-02-25T21:56:15.853 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mgr-diskprediction-local (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:15.864 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:15.937 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T21:56:15.937 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T21:56:16.300 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mgr-cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:16.311 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-mgr-dashboard (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:16.322 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-osd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:16.409 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T21:56:16.409 INFO:teuthology.orchestra.run.trial163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T21:56:16.777 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:16.778 INFO:teuthology.orchestra.run.trial163.stdout:Setting up ceph-volume (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T21:56:16.884 INFO:teuthology.orchestra.run.trial163.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-25T21:56:16.968 INFO:teuthology.orchestra.run.trial163.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-25T21:56:17.394 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:56:17.394 INFO:teuthology.orchestra.run.trial163.stdout:Running kernel seems to be up-to-date. 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout:The processor microcode seems to be up-to-date. 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout:No services need to be restarted. 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout:No containers need to be restarted. 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout:No user sessions are running outdated binaries. 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout: 2026-02-25T21:56:17.395 INFO:teuthology.orchestra.run.trial163.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T21:56:18.197 INFO:teuthology.orchestra.run.trial163.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-25T21:56:18.204 DEBUG:teuthology.parallel:result is None 2026-02-25T21:56:18.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T21:56:18.326 DEBUG:teuthology.orchestra.run.trial163:> dpkg-query -W -f '${Version}' ceph 2026-02-25T21:56:18.336 INFO:teuthology.orchestra.run.trial163.stdout:20.3.0-5305-g0886691e-1jammy 2026-02-25T21:56:18.337 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5305-g0886691e-1jammy 2026-02-25T21:56:18.337 INFO:teuthology.task.install:The correct ceph version 20.3.0-5305-g0886691e-1jammy is installed. 2026-02-25T21:56:18.339 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T21:56:18.339 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:56:18.339 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T21:56:18.387 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T21:56:18.387 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:56:18.387 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T21:56:18.438 DEBUG:teuthology.orchestra.run.trial163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T21:56:18.486 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T21:56:18.486 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:56:18.487 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T21:56:18.534 DEBUG:teuthology.orchestra.run.trial163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T21:56:18.582 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T21:56:18.583 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:56:18.583 DEBUG:teuthology.orchestra.run.trial163:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T21:56:18.630 DEBUG:teuthology.orchestra.run.trial163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T21:56:18.678 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T21:56:18.695 INFO:tasks.workunit:Pulling workunits from ref 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T21:56:18.695 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T21:56:18.695 INFO:tasks.workunit:timeout=3h 2026-02-25T21:56:18.695 INFO:tasks.workunit:cleanup=True 2026-02-25T21:56:18.695 DEBUG:teuthology.orchestra.run.trial163:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T21:56:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T21:56:18.722 INFO:teuthology.orchestra.run.trial163.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T21:56:18.722 DEBUG:teuthology.orchestra.run.trial163:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T21:56:18.766 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T21:56:18.766 DEBUG:teuthology.orchestra.run.trial163:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T21:56:18.811 DEBUG:teuthology.orchestra.run.trial163:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/Ericmzhang/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T21:56:18.862 INFO:tasks.workunit.client.0.trial163.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:Note: switching to '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'. 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: git switch -c 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:Or undo this operation with: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: git switch - 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T21:56:36.218 INFO:tasks.workunit.client.0.trial163.stderr: 2026-02-25T21:56:36.219 INFO:tasks.workunit.client.0.trial163.stderr:HEAD is now at 0886691e8f7 doc/rados/operations/health-checks: Added MON_COLOCATED health warning Update the Health Checks doc page with MON_COLOCATED warning 2026-02-25T21:56:36.220 DEBUG:teuthology.orchestra.run.trial163:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T21:56:36.272 DEBUG:teuthology.orchestra.run.trial163:> set -ex 2026-02-25T21:56:36.272 DEBUG:teuthology.orchestra.run.trial163:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T21:56:36.318 INFO:tasks.workunit:Running workunits matching misc on client.0... 2026-02-25T21:56:36.319 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2026-02-25T21:56:36.319 DEBUG:teuthology.orchestra.run.trial163:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2026-02-25T21:56:36.369 INFO:tasks.workunit.client.0.trial163.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T21:56:36.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/mclock-config 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none ' 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-25T21:56:36.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2026-02-25T21:56:36.375 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2026-02-25T21:56:36.375 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T21:56:36.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:TEST_profile_builtin_to_custom 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:TEST_profile_custom_to_builtin 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:TEST_recovery_limit_adjustment_mclock' 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:36.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:36.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:36.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:36.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:36.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:36.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:36.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:36.382 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:36.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:36.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:36.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:36.384 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:36.385 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:36.386 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:36.386 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:36.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:36.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.387 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:36.388 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:36.388 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:36.388 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:56:36.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:56:36.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:56:36.392 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:36.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:36.396 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:36.396 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:36.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:36.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:36.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:36.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:36.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:36.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:36.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:36.400 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:36.400 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:36.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:36.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:36.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:36.404 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:36.404 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.404 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.404 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:36.405 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:36.405 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:36.405 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:56:36.407 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:56:36.407 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.407 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.407 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:56:36.409 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:56:36.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:56:36.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:56:36.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:56:36.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-25T21:56:36.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T21:56:36.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T21:56:36.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T21:56:36.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T21:56:36.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T21:56:36.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:36.445 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T21:56:36.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:56:36.476 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:56:36.477 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:36.477 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.477 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.477 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:56:36.477 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:56:36.477 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get fsid 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:56:36.534 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:56:36.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get mon_host 2026-02-25T21:56:36.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-25T21:56:36.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T21:56:36.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T21:56:36.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T21:56:36.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T21:56:36.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T21:56:36.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T21:56:36.704 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T21:56:36.704 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:36.704 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:36.704 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:36.704 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.705 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.705 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:36.705 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:56:36.707 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:56:36.727 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T21:56:36.728 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T21:56:36.729 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:56:36.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 2026-02-25T21:56:36.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e7d080e7-eaa2-4750-a317-1fc6e8aff3d4' 2026-02-25T21:56:36.730 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 2026-02-25T21:56:36.730 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:56:36.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAUcJ9pQKYOLBAAKWfDOTZC+LibRdBbFTAboQ== 2026-02-25T21:56:36.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAUcJ9pQKYOLBAAKWfDOTZC+LibRdBbFTAboQ=="}' 2026-02-25T21:56:36.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 -i td/mclock-config/0/new.json 2026-02-25T21:56:36.849 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:36.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T21:56:36.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAUcJ9pQKYOLBAAKWfDOTZC+LibRdBbFTAboQ== --osd-uuid e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 2026-02-25T21:56:36.873 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:36.866+0000 7f33b8286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:36.874 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:36.866+0000 7f33b8286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:36.875 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:36.866+0000 7f33b8286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:36.875 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:36.870+0000 7f33b8286a40 -1 bdev(0x55bdcde46400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T21:56:36.875 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:36.870+0000 7f33b8286a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T21:56:37.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T21:56:37.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:56:37.568 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T21:56:37.568 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T21:56:37.568 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:56:37.679 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:56:37.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T21:56:37.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T21:56:37.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:56:37.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:56:37.680 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:56:37.693 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:37.686+0000 7fd196ce3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:37.695 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:37.686+0000 7fd196ce3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:37.696 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:37.690+0000 7fd196ce3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:37.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:37.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:37.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:38.213 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:38.206+0000 7fd196ce3a40 -1 Falling back to public interface 2026-02-25T21:56:38.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:38.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:38.908 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:56:38.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:56:38.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:38.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:39.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:39.247 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:39.246+0000 7fd196ce3a40 -1 osd.0 0 log_to_monitors true 2026-02-25T21:56:40.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:40.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:40.018 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:56:40.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:56:40.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:40.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:40.234 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2535452358,v1:127.0.0.1:6801/2535452358] [v2:127.0.0.1:6802/2535452358,v1:127.0.0.1:6803/2535452358] exists,up e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 2026-02-25T21:56:40.234 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:56:40.234 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:56:40.234 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:56:40.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:40.235 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:40.236 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:40.236 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:40.236 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T21:56:40.296 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2026-02-25T21:56:40.296 INFO:tasks.workunit.client.0.trial163.stdout:osd_max_backfills: 1 2026-02-25T21:56:40.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2026-02-25T21:56:40.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-25T21:56:40.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-25T21:56:42.519 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T21:56:42.519 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:42.519 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:42.519 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:42.519 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:42.520 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:42.520 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:42.520 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:42.520 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:42.521 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:42.521 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T21:56:42.581 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2026-02-25T21:56:42.581 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-25T21:56:42.581 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-25T21:56:42.581 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:42.582 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:42.582 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:42.582 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:42.582 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:42.582 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:42.582 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:42.583 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:42.583 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:42.583 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T21:56:42.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-25T21:56:42.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-25T21:56:42.643 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-25T21:56:42.643 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:42.644 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:42.644 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:42.644 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:42.644 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:42.644 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:42.644 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:42.645 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:42.645 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:42.645 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T21:56:42.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-25T21:56:42.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-25T21:56:42.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-25T21:56:42.916 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-25T21:56:43.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-25T21:56:45.136 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T21:56:45.136 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:45.137 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:45.137 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:45.137 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:45.137 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:45.137 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:45.137 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:45.138 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:45.138 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:45.138 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T21:56:45.198 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T21:56:45.198 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T21:56:45.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-25T21:56:45.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:45.199 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:45.199 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:45.199 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:45.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:45.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:45.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:45.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:45.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:45.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T21:56:45.259 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T21:56:45.259 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T21:56:45.259 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-25T21:56:45.260 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:45.260 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:45.260 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:45.260 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:45.260 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:45.260 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:45.261 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:45.261 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:45.261 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:45.261 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T21:56:45.322 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:45.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:45.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:45.632 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2026-02-25T21:56:45.845 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 is already down. 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:45.854 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:45.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:56:45.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:45.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/2535452358,v1:127.0.0.1:6801/2535452358] [v2:127.0.0.1:6802/2535452358,v1:127.0.0.1:6803/2535452358] exists e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/mclock-config 2026-02-25T21:56:46.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/mclock-config/0 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:46.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2026-02-25T21:56:46.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/mclock-config/0 2026-02-25T21:56:46.084 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:56:46.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T21:56:46.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2026-02-25T21:56:46.084 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/mclock-config/0/whoami 2026-02-25T21:56:46.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T21:56:46.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T21:56:46.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T21:56:46.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T21:56:46.108 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:46.106+0000 7f6bc4f4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:46.109 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:46.106+0000 7f6bc4f4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:46.110 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:46.106+0000 7f6bc4f4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:46.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:46.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:46.533 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:46.530+0000 7f6bc4f4ca40 -1 Falling back to public interface 2026-02-25T21:56:47.130 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:47.126+0000 7f6bc4f4ca40 -1 osd.0 6 log_to_monitors true 2026-02-25T21:56:47.511 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:47.512 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:56:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:56:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:47.720 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6800/3730893447,v1:127.0.0.1:6801/3730893447] [v2:127.0.0.1:6802/3730893447,v1:127.0.0.1:6803/3730893447] exists,up e7d080e7-eaa2-4750-a317-1fc6e8aff3d4 2026-02-25T21:56:47.720 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:56:47.720 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:56:47.720 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:56:47.721 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T21:56:47.721 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:47.721 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:47.721 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:47.721 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:47.721 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:47.722 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:47.722 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:47.722 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:47.722 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:47.722 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T21:56:47.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T21:56:47.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T21:56:47.782 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-25T21:56:47.782 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:47.782 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:47.782 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:47.782 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:47.783 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:47.783 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:47.783 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:47.783 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:47.783 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:47.783 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:47.841 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:47.842 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:47.842 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:47.842 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:47.842 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:47.842 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:56:47.842 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:47.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:47.901 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:47.901 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:47.901 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:48.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:48.008 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:48.009 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:48.009 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:48.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:48.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:48.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:48.012 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:48.012 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:48.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:48.013 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:48.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:48.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:48.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:48.020 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:48.020 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.020 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:48.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:48.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:48.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T21:56:48.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:48.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:48.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:48.023 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:48.023 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:48.023 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:48.023 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:48.023 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:48.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:48.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:48.026 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:48.027 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:48.028 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:48.028 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:48.028 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:48.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:48.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:48.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:48.030 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:48.031 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:48.032 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:48.032 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:48.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:48.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:48.035 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:48.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:48.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:48.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:48.038 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:48.038 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:48.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:48.039 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:48.041 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:48.041 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:48.041 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:48.042 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:48.042 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:48.043 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:48.043 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:48.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:48.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:48.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:48.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:48.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:48.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:48.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:48.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:56:48.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:56:48.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T21:56:48.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T21:56:48.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T21:56:48.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T21:56:48.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T21:56:48.073 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:48.074 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T21:56:48.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T21:56:48.104 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T21:56:48.104 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:56:48.104 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:56:48.104 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.105 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.106 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:56:48.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:56:48.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get fsid 2026-02-25T21:56:48.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T21:56:48.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:56:48.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:56:48.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.174 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:56:48.175 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:56:48.175 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get mon_host 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T21:56:48.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.346 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:48.347 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:56:48.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:56:48.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T21:56:48.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T21:56:48.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:56:48.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T21:56:48.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:48.366 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T21:56:48.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T21:56:48.368 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:56:48.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=58aaef7f-b92d-4a15-8c23-0d2d4e9ab8d6 2026-02-25T21:56:48.368 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 58aaef7f-b92d-4a15-8c23-0d2d4e9ab8d6 2026-02-25T21:56:48.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 58aaef7f-b92d-4a15-8c23-0d2d4e9ab8d6' 2026-02-25T21:56:48.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:56:48.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAgcJ9p94J1FhAA1hiGcpFGfTADAkfya/z7yA== 2026-02-25T21:56:48.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAgcJ9p94J1FhAA1hiGcpFGfTADAkfya/z7yA=="}' 2026-02-25T21:56:48.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 58aaef7f-b92d-4a15-8c23-0d2d4e9ab8d6 -i td/mclock-config/0/new.json 2026-02-25T21:56:48.488 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:48.496 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T21:56:48.497 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAgcJ9p94J1FhAA1hiGcpFGfTADAkfya/z7yA== --osd-uuid 58aaef7f-b92d-4a15-8c23-0d2d4e9ab8d6 2026-02-25T21:56:48.512 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:48.510+0000 7f675cfc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:48.514 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:48.510+0000 7f675cfc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:48.514 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:48.510+0000 7f675cfc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:48.515 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:48.514+0000 7f675cfc8a40 -1 bdev(0x55906cae8400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T21:56:48.515 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:48.514+0000 7f675cfc8a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T21:56:49.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T21:56:49.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:56:49.348 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T21:56:49.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T21:56:49.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:56:49.473 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:56:49.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T21:56:49.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T21:56:49.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:56:49.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:56:49.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:56:49.495 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:49.494+0000 7f8b5463aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:49.497 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:49.494+0000 7f8b5463aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:49.498 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:49.494+0000 7f8b5463aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:49.584 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:49.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:56:49.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:49.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:49.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:49.908 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:49.906+0000 7f8b5463aa40 -1 Falling back to public interface 2026-02-25T21:56:50.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:50.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:50.697 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:56:50.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:56:50.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:50.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:50.856 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:50.854+0000 7f8b5463aa40 -1 osd.0 0 log_to_monitors true 2026-02-25T21:56:50.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:51.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:51.909 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:56:51.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:51.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:56:51.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:51.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:52.124 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/251019610,v1:127.0.0.1:6801/251019610] [v2:127.0.0.1:6802/251019610,v1:127.0.0.1:6803/251019610] exists,up 58aaef7f-b92d-4a15-8c23-0d2d4e9ab8d6 2026-02-25T21:56:52.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:56:52.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:56:52.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:56:52.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2026-02-25T21:56:52.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2026-02-25T21:56:52.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2026-02-25T21:56:52.338 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-25T21:56:52.339 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T21:56:52.339 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:52.339 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:52.339 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:52.339 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:52.339 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:52.340 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:52.340 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T21:56:52.340 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T21:56:52.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2026-02-25T21:56:52.401 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2026-02-25T21:56:52.401 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2026-02-25T21:56:52.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2026-02-25T21:56:52.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2026-02-25T21:56:52.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-25T21:56:52.484 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:56:52.484 INFO:tasks.workunit.client.0.trial163.stdout: "success": "" 2026-02-25T21:56:52.484 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:56:52.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:52.493 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:52.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T21:56:52.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T21:56:52.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2026-02-25T21:56:52.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2026-02-25T21:56:52.555 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2026-02-25T21:56:52.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2026-02-25T21:56:52.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2026-02-25T21:56:52.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-25T21:56:52.556 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2026-02-25T21:56:52.556 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T21:56:52.556 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:52.556 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:52.556 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:52.556 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:52.557 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:52.557 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:52.557 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T21:56:52.557 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T21:56:52.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.000000 2026-02-25T21:56:52.615 INFO:tasks.workunit.client.0.trial163.stdout:client_res = 0.000000 2026-02-25T21:56:52.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.000000' 2026-02-25T21:56:52.616 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.000000 + 0.1' 2026-02-25T21:56:52.616 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2026-02-25T21:56:52.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.100000 2026-02-25T21:56:52.617 INFO:tasks.workunit.client.0.trial163.stdout:client_res_new = .100000 2026-02-25T21:56:52.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .100000' 2026-02-25T21:56:52.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .100000 2026-02-25T21:56:52.828 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T21:56:53.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.100000 2026-02-25T21:56:53.052 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.100000 != .100000' 2026-02-25T21:56:53.052 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2026-02-25T21:56:53.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-25T21:56:53.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:53.055 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T21:56:53.056 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T21:56:53.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.100000 2026-02-25T21:56:53.119 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.100000 != .100000' 2026-02-25T21:56:53.119 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:53.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:53.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:53.229 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:53.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:53.230 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:53.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:53.232 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:53.232 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:53.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:53.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:53.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.235 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:53.236 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:53.236 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:53.241 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:53.241 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.241 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.241 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:53.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:53.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:53.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:53.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:53.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:53.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:53.249 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:53.249 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:53.249 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:53.250 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.250 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:53.251 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:53.251 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.252 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:53.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:53.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:53.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:53.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:53.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:53.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:53.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:53.258 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:53.258 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:53.259 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:53.259 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:53.260 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:53.260 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:53.260 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:53.261 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.261 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:53.262 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:53.262 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:53.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:53.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:53.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:53.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.266 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:53.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:53.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:53.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:56:53.268 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:56:53.268 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.269 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:56:53.270 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:56:53.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:56:53.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:56:53.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:56:53.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:56:53.274 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:56:53.275 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:56:53.275 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:56:53.276 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:56:53.276 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:56:53.276 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:56:53.277 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:56:53.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:56:53.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:56:53.279 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:56:53.280 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:56:53.280 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:56:53.280 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:56:53.282 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:56:53.282 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.282 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.282 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:56:53.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:56:53.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:56:53.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:56:53.284 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:56:53.285 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.285 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.285 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T21:56:53.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T21:56:53.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T21:56:53.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T21:56:53.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T21:56:53.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T21:56:53.309 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T21:56:53.309 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:53.309 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:53.309 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:53.309 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.309 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.310 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:53.310 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T21:56:53.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T21:56:53.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T21:56:53.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:56:53.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:56:53.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.340 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.341 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:56:53.341 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:56:53.341 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get fsid 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:56:53.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:56:53.399 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:53.399 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.399 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.399 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:56:53.399 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:56:53.399 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get mon_host 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T21:56:53.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:53.566 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:56:53.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:56:53.585 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T21:56:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T21:56:53.587 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:56:53.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2b0c58cc-271d-42f6-9b18-f40a3543b2b8 2026-02-25T21:56:53.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2b0c58cc-271d-42f6-9b18-f40a3543b2b8' 2026-02-25T21:56:53.588 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 2b0c58cc-271d-42f6-9b18-f40a3543b2b8 2026-02-25T21:56:53.588 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:56:53.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlcJ9p3iSWIxAAS5GO+1XCn7jC5+syRHaj5A== 2026-02-25T21:56:53.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlcJ9p3iSWIxAAS5GO+1XCn7jC5+syRHaj5A=="}' 2026-02-25T21:56:53.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2b0c58cc-271d-42f6-9b18-f40a3543b2b8 -i td/mclock-config/0/new.json 2026-02-25T21:56:53.708 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:53.715 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T21:56:53.717 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAlcJ9p3iSWIxAAS5GO+1XCn7jC5+syRHaj5A== --osd-uuid 2b0c58cc-271d-42f6-9b18-f40a3543b2b8 2026-02-25T21:56:53.732 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:53.730+0000 7f85539a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:53.733 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:53.730+0000 7f85539a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:53.734 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:53.730+0000 7f85539a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:53.735 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:53.730+0000 7f85539a7a40 -1 bdev(0x564cb27a8400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T21:56:53.735 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:53.734+0000 7f85539a7a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T21:56:54.451 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T21:56:54.451 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:56:54.452 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T21:56:54.453 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T21:56:54.453 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:56:54.577 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:56:54.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T21:56:54.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T21:56:54.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:56:54.578 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:56:54.578 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:56:54.600 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:54.598+0000 7f0a202a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:54.601 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:54.598+0000 7f0a202a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:54.603 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:54.598+0000 7f0a202a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:54.688 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:56:54.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:56:54.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:54.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:54.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:55.029 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:55.026+0000 7f0a202a0a40 -1 Falling back to public interface 2026-02-25T21:56:55.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:55.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:55.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:56:55.796 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:56:55.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:55.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:55.905 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:56:56.036 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:56:56.034+0000 7f0a202a0a40 -1 osd.0 0 log_to_monitors true 2026-02-25T21:56:56.907 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:56:56.907 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:56:56.907 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:56:56.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:56:56.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:56:56.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:56:57.122 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3328419805,v1:127.0.0.1:6801/3328419805] [v2:127.0.0.1:6802/3328419805,v1:127.0.0.1:6803/3328419805] exists,up 2b0c58cc-271d-42f6-9b18-f40a3543b2b8 2026-02-25T21:56:57.123 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:56:57.123 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:56:57.123 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:56:57.123 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2026-02-25T21:56:57.123 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2026-02-25T21:56:57.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2026-02-25T21:56:57.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2026-02-25T21:56:57.335 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-25T21:56:57.335 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:56:57.335 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:57.335 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:57.336 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:57.336 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:57.336 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:57.336 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:57.336 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:56:57.336 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T21:56:57.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2026-02-25T21:56:57.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2026-02-25T21:56:57.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2026-02-25T21:56:57.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2026-02-25T21:56:57.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-25T21:56:57.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-25T21:56:57.488 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:56:57.488 INFO:tasks.workunit.client.0.trial163.stdout: "success": "" 2026-02-25T21:56:57.488 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:56:57.496 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-25T21:56:57.496 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:56:57.497 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T21:56:57.557 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2026-02-25T21:56:57.559 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2026-02-25T21:56:57.559 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2026-02-25T21:56:57.559 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2026-02-25T21:56:57.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2026-02-25T21:56:57.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2026-02-25T21:56:57.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T21:56:57.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2026-02-25T21:56:57.561 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:56:57.561 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:57.561 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:57.561 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:57.561 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:57.561 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:57.562 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:57.562 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:56:57.562 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T21:56:57.622 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.000000 2026-02-25T21:56:57.622 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.000000' 2026-02-25T21:56:57.622 INFO:tasks.workunit.client.0.trial163.stdout:Original client_res for osd.0 = 0.000000 2026-02-25T21:56:57.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.000000 + 0.1' 2026-02-25T21:56:57.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:56:57.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.100000 2026-02-25T21:56:57.624 INFO:tasks.workunit.client.0.trial163.stdout:client_res_new = .100000 2026-02-25T21:56:57.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .100000' 2026-02-25T21:56:57.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .100000 2026-02-25T21:56:57.839 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T21:56:58.070 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.100000 2026-02-25T21:56:58.071 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T21:56:58.071 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:56:58.072 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T21:56:58.073 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T21:56:58.073 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2026-02-25T21:56:58.073 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:56:58.073 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:58.073 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:58.074 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:58.074 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:58.074 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:58.074 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:58.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:56:58.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T21:56:58.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.100000 2026-02-25T21:56:58.135 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T21:56:58.135 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:56:58.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T21:56:58.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2026-02-25T21:56:58.212 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:56:58.212 INFO:tasks.workunit.client.0.trial163.stdout: "success": "" 2026-02-25T21:56:58.212 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:56:58.220 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-25T21:56:58.220 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:56:58.220 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:58.220 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:58.221 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:58.221 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:58.221 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:58.221 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:58.221 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:56:58.221 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T21:56:58.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2026-02-25T21:56:58.283 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2026-02-25T21:56:58.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-25T21:56:58.284 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T21:56:58.496 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.100000 2026-02-25T21:56:58.497 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T21:56:58.497 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:56:58.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T21:56:58.499 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T21:56:58.499 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:56:58.500 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:56:58.501 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:56:58.501 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T21:56:58.559 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.100000 2026-02-25T21:56:58.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T21:56:58.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:56:58.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T21:56:58.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2026-02-25T21:56:58.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2026-02-25T21:57:03.777 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T21:57:03.984 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2026-02-25T21:57:03.985 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2026-02-25T21:57:03.985 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:57:03.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T21:57:03.987 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T21:57:03.987 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2026-02-25T21:57:03.987 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T21:57:03.987 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:57:03.987 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:57:03.988 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:03.988 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:03.988 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:03.988 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:57:03.988 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T21:57:03.989 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T21:57:04.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.000000 2026-02-25T21:57:04.048 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.000000 != 0.000000' 2026-02-25T21:57:04.048 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2026-02-25T21:57:04.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:04.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:04.158 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:04.158 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:04.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:04.159 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:04.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:04.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:04.160 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:04.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:04.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:04.162 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:04.164 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:04.165 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:04.170 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:04.170 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.170 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:04.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:04.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:04.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T21:57:04.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:04.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:04.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:04.172 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:04.172 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:04.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:04.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:04.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:04.175 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:04.175 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:04.176 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:04.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:04.177 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:04.177 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:04.177 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:04.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:04.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:04.179 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:04.181 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:04.181 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:04.182 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:04.182 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.182 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.183 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:04.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:04.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:04.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T21:57:04.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:04.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:04.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:04.188 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:04.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:04.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:04.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:04.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:04.190 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:04.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:04.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:04.192 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:04.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:04.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:04.195 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:04.195 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.195 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:04.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:04.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:04.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:57:04.198 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:57:04.198 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.198 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.198 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:57:04.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:57:04.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:57:04.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:04.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:04.202 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:04.203 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:04.204 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:04.204 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:04.205 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:04.205 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:04.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:04.206 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:04.207 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.207 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:04.207 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:04.208 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:04.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:04.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:04.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:04.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:04.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:04.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:04.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:04.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T21:57:04.214 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:57:04.214 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.214 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11492 2026-02-25T21:57:04.215 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T21:57:04.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.238 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:57:04.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T21:57:04.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T21:57:04.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T21:57:04.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:57:04.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:57:04.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T21:57:04.268 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T21:57:04.268 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:57:04.268 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:57:04.268 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:57:04.269 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:04.269 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.269 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.269 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:57:04.269 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:57:04.269 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get fsid 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:04.326 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.327 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.327 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-mon.a.asok 2026-02-25T21:57:04.327 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:57:04.327 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11492/ceph-mon.a.asok config get mon_host 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T21:57:04.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:57:04.496 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:57:04.497 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:57:04.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T21:57:04.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:04.516 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T21:57:04.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T21:57:04.518 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:57:04.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5b3fc35b-b8ff-46f8-adb1-fa5ae3303b7c 2026-02-25T21:57:04.519 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 5b3fc35b-b8ff-46f8-adb1-fa5ae3303b7c 2026-02-25T21:57:04.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5b3fc35b-b8ff-46f8-adb1-fa5ae3303b7c' 2026-02-25T21:57:04.519 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:57:04.531 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAwcJ9pLnmTHxAA+WJg7BS0J/luI68cAXaK8w== 2026-02-25T21:57:04.531 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAwcJ9pLnmTHxAA+WJg7BS0J/luI68cAXaK8w=="}' 2026-02-25T21:57:04.531 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5b3fc35b-b8ff-46f8-adb1-fa5ae3303b7c -i td/mclock-config/0/new.json 2026-02-25T21:57:04.641 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:57:04.648 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T21:57:04.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAwcJ9pLnmTHxAA+WJg7BS0J/luI68cAXaK8w== --osd-uuid 5b3fc35b-b8ff-46f8-adb1-fa5ae3303b7c 2026-02-25T21:57:04.665 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:04.661+0000 7f03de0eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:04.666 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:04.661+0000 7f03de0eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:04.667 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:04.661+0000 7f03de0eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:04.667 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:04.665+0000 7f03de0eba40 -1 bdev(0x55d32592c400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T21:57:04.667 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:04.665+0000 7f03de0eba40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T21:57:05.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T21:57:05.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:57:05.436 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T21:57:05.436 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T21:57:05.437 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:57:05.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T21:57:05.560 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:57:05.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=5ef79763-5a43-4e8e-9ae2-b24978ac122d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11492/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T21:57:05.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:57:05.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:57:05.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:57:05.582 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:05.577+0000 7fda9df66a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:05.584 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:05.581+0000 7fda9df66a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:05.585 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:05.581+0000 7fda9df66a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:05.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:57:05.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:57:06.085 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:06.081+0000 7fda9df66a40 -1 Falling back to public interface 2026-02-25T21:57:06.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:57:06.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:06.787 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:57:06.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:57:06.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:06.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:57:06.991 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:57:07.031 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:07.025+0000 7fda9df66a40 -1 osd.0 0 log_to_monitors true 2026-02-25T21:57:07.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:57:07.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:07.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:57:07.993 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:57:07.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:07.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:57:08.199 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/190633083,v1:127.0.0.1:6801/190633083] [v2:127.0.0.1:6802/190633083,v1:127.0.0.1:6803/190633083] exists,up 5b3fc35b-b8ff-46f8-adb1-fa5ae3303b7c 2026-02-25T21:57:08.199 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:57:08.199 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:57:08.199 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:57:08.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:57:08.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:57:08.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:57:08.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:08.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:08.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:08.201 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:57:08.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:57:08.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-25T21:57:08.260 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2026-02-25T21:57:08.260 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2026-02-25T21:57:08.260 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2026-02-25T21:57:08.261 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-25T21:57:08.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-25T21:57:10.482 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:57:10.482 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:57:10.482 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:57:10.483 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:10.483 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:10.483 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:10.483 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:57:10.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:57:10.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-25T21:57:10.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2026-02-25T21:57:10.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2026-02-25T21:57:10.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-25T21:57:10.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-25T21:57:11.007 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-25T21:57:13.009 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T21:57:13.009 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:57:13.009 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:57:13.010 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:13.010 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.010 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:13.010 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11492/ceph-osd.0.asok 2026-02-25T21:57:13.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T21:57:13.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11492/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-25T21:57:13.071 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2026-02-25T21:57:13.071 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2026-02-25T21:57:13.071 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2026-02-25T21:57:13.071 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:13.072 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:13.072 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:13.072 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:13.072 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:13.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:13.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:13.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:13.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:13.180 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:13.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:13.181 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:13.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:13.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:13.182 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:13.183 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.183 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:13.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:13.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:13.186 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:13.186 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:13.191 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:13.191 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.191 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:13.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:13.193 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:13.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:13.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:13.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:13.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:13.196 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:13.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:13.197 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:13.198 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:13.199 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:13.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:13.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:13.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:13.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.202 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:13.203 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:13.203 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:13.204 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:13.204 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.204 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:13.204 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/mclock-config 0 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:13.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:13.207 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:13.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:13.209 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:13.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:13.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:13.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:13.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:13.212 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:13.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:13.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:13.214 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:13.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T21:57:13.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T21:57:13.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:13.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11492 2026-02-25T21:57:13.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11492 2026-02-25T21:57:13.219 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:13.219 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:13.219 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T21:57:13.219 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T21:57:13.219 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T21:57:13.275 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2026-02-25T21:57:13.275 DEBUG:teuthology.orchestra.run.trial163:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2026-02-25T21:57:13.324 INFO:tasks.workunit.client.0.trial163.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/network-ping 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2026-02-25T21:57:13.327 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2026-02-25T21:57:13.328 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none ' 2026-02-25T21:57:13.328 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2026-02-25T21:57:13.328 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2026-02-25T21:57:13.328 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2026-02-25T21:57:13.328 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2026-02-25T21:57:13.329 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2026-02-25T21:57:13.329 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:TEST_network_ping_test2' 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:57:13.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T21:57:13.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:57:13.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:57:13.332 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:57:13.332 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:57:13.332 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:57:13.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:57:13.334 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:57:13.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:57:13.335 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:57:13.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:57:13.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:57:13.337 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:57:13.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:57:13.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:57:13.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:57:13.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:57:13.341 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:57:13.341 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T21:57:13.342 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:57:13.342 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.342 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.343 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.23896 2026-02-25T21:57:13.344 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:57:13.344 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:57:13.344 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-25T21:57:13.345 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:57:13.345 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.345 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.345 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.23896 2026-02-25T21:57:13.347 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-25T21:57:13.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-25T21:57:13.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T21:57:13.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:13.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:13.371 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:13.371 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.371 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.372 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:13.372 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T21:57:13.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:57:13.403 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mon.a.asok 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:57:13.404 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.23896/ceph-mon.a.asok config get fsid 2026-02-25T21:57:13.461 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T21:57:13.461 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:57:13.461 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:57:13.461 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T21:57:13.461 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mon.a.asok 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:57:13.462 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.23896/ceph-mon.a.asok config get mon_host 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-25T21:57:13.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:13.633 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:57:13.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:13.654 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-25T21:57:13.655 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:57:13.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=62ab8481-1785-40bc-a248-34426e170b0e 2026-02-25T21:57:13.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 62ab8481-1785-40bc-a248-34426e170b0e' 2026-02-25T21:57:13.657 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 62ab8481-1785-40bc-a248-34426e170b0e 2026-02-25T21:57:13.657 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:57:13.668 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5cJ9p1BS9JxAAW5loOCo2ABmTuDwUVjVMlQ== 2026-02-25T21:57:13.668 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5cJ9p1BS9JxAAW5loOCo2ABmTuDwUVjVMlQ=="}' 2026-02-25T21:57:13.668 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 62ab8481-1785-40bc-a248-34426e170b0e -i td/network-ping/0/new.json 2026-02-25T21:57:13.777 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:57:13.784 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-25T21:57:13.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5cJ9p1BS9JxAAW5loOCo2ABmTuDwUVjVMlQ== --osd-uuid 62ab8481-1785-40bc-a248-34426e170b0e 2026-02-25T21:57:13.801 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:13.797+0000 7f194c798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:13.802 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:13.797+0000 7f194c798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:13.803 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:13.801+0000 7f194c798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:13.803 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:13.801+0000 7f194c798a40 -1 bdev(0x560cef068400 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-25T21:57:13.804 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:13.801+0000 7f194c798a40 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-25T21:57:14.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-25T21:57:14.479 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:57:14.479 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T21:57:14.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T21:57:14.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:57:14.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T21:57:14.603 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:57:14.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T21:57:14.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:57:14.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:57:14.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:57:14.628 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:14.625+0000 7f6cf024da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:14.631 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:14.625+0000 7f6cf024da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:14.632 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:14.629+0000 7f6cf024da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:14.712 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:14.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:57:14.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:57:15.053 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:15.049+0000 7f6cf024da40 -1 Falling back to public interface 2026-02-25T21:57:15.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:57:15.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:15.822 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:57:15.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:57:15.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:15.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:57:15.931 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:57:16.072 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:16.069+0000 7f6cf024da40 -1 osd.0 0 log_to_monitors true 2026-02-25T21:57:16.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:57:16.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:16.933 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:57:16.934 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:57:16.934 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:16.934 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:57:17.144 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2437994953,v1:127.0.0.1:6801/2437994953] [v2:127.0.0.1:6802/2437994953,v1:127.0.0.1:6803/2437994953] exists,up 62ab8481-1785-40bc-a248-34426e170b0e 2026-02-25T21:57:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:57:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-25T21:57:17.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T21:57:17.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:57:17.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1e27c11a-2bdd-48fa-abaf-3454cb834899 2026-02-25T21:57:17.150 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 1e27c11a-2bdd-48fa-abaf-3454cb834899 2026-02-25T21:57:17.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1e27c11a-2bdd-48fa-abaf-3454cb834899' 2026-02-25T21:57:17.150 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:57:17.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA9cJ9pSQZ8CRAAOgPpUTHdhQLmS90R9yTBrQ== 2026-02-25T21:57:17.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA9cJ9pSQZ8CRAAOgPpUTHdhQLmS90R9yTBrQ=="}' 2026-02-25T21:57:17.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1e27c11a-2bdd-48fa-abaf-3454cb834899 -i td/network-ping/1/new.json 2026-02-25T21:57:17.379 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:57:17.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-25T21:57:17.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9cJ9pSQZ8CRAAOgPpUTHdhQLmS90R9yTBrQ== --osd-uuid 1e27c11a-2bdd-48fa-abaf-3454cb834899 2026-02-25T21:57:17.407 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:17.405+0000 7f9751c85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:17.409 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:17.405+0000 7f9751c85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:17.410 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:17.405+0000 7f9751c85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:17.410 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:17.405+0000 7f9751c85a40 -1 bdev(0x5601b1e00400 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-25T21:57:17.410 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:17.405+0000 7f9751c85a40 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-25T21:57:18.002 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-25T21:57:18.002 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:57:18.002 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T21:57:18.002 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T21:57:18.003 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:57:18.290 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T21:57:18.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T21:57:18.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T21:57:18.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:57:18.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:57:18.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:57:18.315 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:18.309+0000 7f5c57081a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:18.316 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:18.313+0000 7f5c57081a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:18.317 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:18.313+0000 7f5c57081a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:57:18.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:18.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T21:57:18.717 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:18.713+0000 7f5c57081a40 -1 Falling back to public interface 2026-02-25T21:57:18.717 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:57:19.276 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:19.273+0000 7f5c57081a40 -1 osd.1 0 log_to_monitors true 2026-02-25T21:57:19.718 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:57:19.718 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:19.719 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:57:19.719 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:57:19.719 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:19.719 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T21:57:19.924 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/669834081,v1:127.0.0.1:6811/669834081] [v2:127.0.0.1:6812/669834081,v1:127.0.0.1:6813/669834081] exists,up 1e27c11a-2bdd-48fa-abaf-3454cb834899 2026-02-25T21:57:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:57:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:57:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:57:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2026-02-25T21:57:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T21:57:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:57:19.926 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T21:57:19.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bf0bdaee-c6a5-4d1b-a86f-071f1b93ce19 2026-02-25T21:57:19.928 INFO:tasks.workunit.client.0.trial163.stdout:add osd2 bf0bdaee-c6a5-4d1b-a86f-071f1b93ce19 2026-02-25T21:57:19.929 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 bf0bdaee-c6a5-4d1b-a86f-071f1b93ce19' 2026-02-25T21:57:19.929 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:57:19.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA/cJ9pYQb/NxAAzruNguvR6oo8OpkVuSG3tQ== 2026-02-25T21:57:19.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA/cJ9pYQb/NxAAzruNguvR6oo8OpkVuSG3tQ=="}' 2026-02-25T21:57:19.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bf0bdaee-c6a5-4d1b-a86f-071f1b93ce19 -i td/network-ping/2/new.json 2026-02-25T21:57:20.167 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:57:20.176 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-25T21:57:20.177 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/cJ9pYQb/NxAAzruNguvR6oo8OpkVuSG3tQ== --osd-uuid bf0bdaee-c6a5-4d1b-a86f-071f1b93ce19 2026-02-25T21:57:20.194 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:20.189+0000 7f3d2f32ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:20.195 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:20.193+0000 7f3d2f32ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:20.196 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:20.193+0000 7f3d2f32ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:20.197 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:20.193+0000 7f3d2f32ca40 -1 bdev(0x55688a176400 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-25T21:57:20.197 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:20.193+0000 7f3d2f32ca40 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-25T21:57:20.841 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-25T21:57:20.842 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:57:20.842 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T21:57:20.843 INFO:tasks.workunit.client.0.trial163.stdout:adding osd2 key to auth repository 2026-02-25T21:57:20.843 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:57:21.142 INFO:tasks.workunit.client.0.trial163.stdout:start osd.2 2026-02-25T21:57:21.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T21:57:21.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T21:57:21.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:57:21.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:57:21.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:57:21.167 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:21.165+0000 7fbcbd71da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:21.169 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:21.165+0000 7fbcbd71da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:21.170 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:21.165+0000 7fbcbd71da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:21.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T21:57:21.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:57:21.569 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:21.565+0000 7fbcbd71da40 -1 Falling back to public interface 2026-02-25T21:57:22.153 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:57:22.149+0000 7fbcbd71da40 -1 osd.2 0 log_to_monitors true 2026-02-25T21:57:22.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:57:22.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:57:22.567 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:57:22.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:57:22.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:57:22.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T21:57:22.777 INFO:tasks.workunit.client.0.trial163.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1672357216,v1:127.0.0.1:6819/1672357216] [v2:127.0.0.1:6820/1672357216,v1:127.0.0.1:6821/1672357216] exists,up bf0bdaee-c6a5-4d1b-a86f-071f1b93ce19 2026-02-25T21:57:22.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:57:22.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:57:22.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:57:22.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2026-02-25T21:57:27.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2026-02-25T21:57:27.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2026-02-25T21:57:28.072 INFO:tasks.workunit.client.0.trial163.stderr:pool 'foo' already exists 2026-02-25T21:57:28.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T21:57:29.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-25T21:57:29.104 INFO:tasks.workunit.client.0.trial163.stdout:hints = 1 2026-02-25T21:57:29.104 INFO:tasks.workunit.client.0.trial163.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-25T21:57:29.104 INFO:tasks.workunit.client.0.trial163.stdout:Object prefix: benchmark_data_trial163_29487 2026-02-25T21:57:29.104 INFO:tasks.workunit.client.0.trial163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-25T21:57:29.105 INFO:tasks.workunit.client.0.trial163.stdout: 0 0 0 0 0 0 - 0 2026-02-25T21:57:30.103 INFO:tasks.workunit.client.0.trial163.stdout: 1 16 10578 10562 41.2546 41.2578 0.00112619 0.00151227 2026-02-25T21:57:31.104 INFO:tasks.workunit.client.0.trial163.stdout: 2 16 21285 21269 41.5371 41.8242 0.00114387 0.00150273 2026-02-25T21:57:32.104 INFO:tasks.workunit.client.0.trial163.stdout: 3 16 32356 32340 42.1056 43.2461 0.000921623 0.00148261 2026-02-25T21:57:33.104 INFO:tasks.workunit.client.0.trial163.stdout: 4 16 43338 43322 42.303 42.8984 0.000996553 0.00147583 2026-02-25T21:57:34.104 INFO:tasks.workunit.client.0.trial163.stdout: 5 16 54074 54058 42.2292 41.9375 0.00139991 0.00147837 2026-02-25T21:57:35.104 INFO:tasks.workunit.client.0.trial163.stdout: 6 16 65103 65087 42.3708 43.082 0.000964624 0.00147348 2026-02-25T21:57:36.104 INFO:tasks.workunit.client.0.trial163.stdout: 7 16 76091 76075 42.449 42.9219 0.00122527 0.00147083 2026-02-25T21:57:37.104 INFO:tasks.workunit.client.0.trial163.stdout: 8 16 86549 86533 42.249 40.8516 0.001418 0.00147775 2026-02-25T21:57:38.104 INFO:tasks.workunit.client.0.trial163.stdout: 9 16 96168 96152 41.7292 37.5742 0.00189774 0.00149619 2026-02-25T21:57:39.104 INFO:tasks.workunit.client.0.trial163.stdout: 10 10 105547 105537 41.222 36.6602 0.00197677 0.00151465 2026-02-25T21:57:40.104 INFO:tasks.workunit.client.0.trial163.stdout:Total time run: 10.0014 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Total writes made: 105547 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Write size: 4096 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Object size: 4096 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Bandwidth (MB/sec): 41.2236 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Stddev Bandwidth: 2.31868 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Max bandwidth (MB/sec): 43.2461 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Min bandwidth (MB/sec): 36.6602 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Average IOPS: 10553 2026-02-25T21:57:40.105 INFO:tasks.workunit.client.0.trial163.stdout:Stddev IOPS: 593.582 2026-02-25T21:57:40.106 INFO:tasks.workunit.client.0.trial163.stdout:Max IOPS: 11071 2026-02-25T21:57:40.106 INFO:tasks.workunit.client.0.trial163.stdout:Min IOPS: 9385 2026-02-25T21:57:40.106 INFO:tasks.workunit.client.0.trial163.stdout:Average Latency(s): 0.00151465 2026-02-25T21:57:40.106 INFO:tasks.workunit.client.0.trial163.stdout:Stddev Latency(s): 0.000723021 2026-02-25T21:57:40.106 INFO:tasks.workunit.client.0.trial163.stdout:Max latency(s): 0.00914638 2026-02-25T21:57:40.106 INFO:tasks.workunit.client.0.trial163.stdout:Min latency(s): 0.000475664 2026-02-25T21:57:40.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2026-02-25T21:57:50.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2026-02-25T21:57:50.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T21:57:50.112 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T21:57:50.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T21:57:50.320 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T21:57:50.320 INFO:tasks.workunit.client.0.trial163.stderr:2' 2026-02-25T21:57:50.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T21:57:50.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:57:50.321 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T21:57:50.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-25T21:57:50.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-25T21:57:50.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836489' 2026-02-25T21:57:50.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:57:50.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T21:57:50.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705673 2026-02-25T21:57:50.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705673 2026-02-25T21:57:50.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836489 1-38654705673' 2026-02-25T21:57:50.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:57:50.473 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T21:57:50.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574856 2026-02-25T21:57:50.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574856 2026-02-25T21:57:50.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836489 1-38654705673 2-55834574856' 2026-02-25T21:57:50.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:57:50.549 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836489 2026-02-25T21:57:50.549 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:57:50.550 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T21:57:50.551 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-25T21:57:50.551 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:57:50.552 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836489 2026-02-25T21:57:50.552 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836489 2026-02-25T21:57:50.552 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-25T21:57:50.553 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:57:50.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-25T21:57:50.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T21:57:51.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T21:57:51.765 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:57:51.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-25T21:57:51.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:57:51.978 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705673 2026-02-25T21:57:51.979 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:57:51.980 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T21:57:51.981 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705673 2026-02-25T21:57:51.981 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:57:51.982 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705673 2026-02-25T21:57:51.982 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 38654705673 2026-02-25T21:57:51.983 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2026-02-25T21:57:51.983 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T21:57:52.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705673 -lt 38654705673 2026-02-25T21:57:52.200 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:57:52.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574856 2026-02-25T21:57:52.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:57:52.202 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T21:57:52.203 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574856 2026-02-25T21:57:52.203 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:57:52.204 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574856 2026-02-25T21:57:52.204 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 55834574856 2026-02-25T21:57:52.205 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574856' 2026-02-25T21:57:52.205 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T21:57:52.874 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574856 -lt 55834574856 2026-02-25T21:57:52.875 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:57:52.875 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:57:52.875 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:57:52.875 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:57:52.875 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:52.876 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:52.876 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:52.876 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:57:52.876 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:57:52.876 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 2026-02-25T21:57:53.037 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:57:53.037 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 1000, 2026-02-25T21:57:53.037 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:57:53.037 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:57:53.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:57:53.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:57:53.056 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.057 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2026-02-25T21:57:53.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T21:57:53.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:57:53.067 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:57:53.067 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:57:53.068 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:57:53.068 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:53.069 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:53.069 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:53.069 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:57:53.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:57:53.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 2026-02-25T21:57:53.124 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:57:53.124 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 1000, 2026-02-25T21:57:53.124 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:57:53.124 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:57:53.178 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.178 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:57:53.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:57:53.188 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2026-02-25T21:57:53.198 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T21:57:53.199 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:53.200 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:57:53.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:57:53.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 0 2026-02-25T21:57:53.623 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 0, 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [ 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:52 2026", 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:57:53.624 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.170, 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.358, 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.304 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.238, 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 3.982, 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 3.982 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.625 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.238, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 3.982, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 3.982 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.237 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:53 2026", 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.626 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.199, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.190, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.250 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.283, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.301, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.554 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.283, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.301, 2026-02-25T21:57:53.627 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.554 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.206 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:53 2026", 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.189, 2026-02-25T21:57:53.628 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.171, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.247 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.259, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.354, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.508 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.259, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.354, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.508 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.259 2026-02-25T21:57:53.629 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:52 2026", 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.175, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.174, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.192 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.630 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.323, 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.335, 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.426 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.323, 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.335, 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.426 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.182 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: } 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout: ] 2026-02-25T21:57:53.631 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:57:53.632 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.632 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:57:53.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2026-02-25T21:57:53.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2026-02-25T21:57:53.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:57:53.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:57:53.804 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:57:53.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:57:53.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 0 2026-02-25T21:57:53.854 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:57:53.854 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 0, 2026-02-25T21:57:53.854 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [ 2026-02-25T21:57:53.854 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.854 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:47 2026", 2026-02-25T21:57:53.854 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.149, 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.380, 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.311 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.855 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.134, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.128, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.128 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.174, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 3.982, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 3.982 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.129 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.856 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.155, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.206, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.256 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.141, 2026-02-25T21:57:53.857 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.113, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.113 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.166, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.554, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.554 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.138 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.858 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.142, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.190, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.254 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.116, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.116, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.116 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.859 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.196, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.508, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.508 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.146 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:50 2026", 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.860 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.144, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.173, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.239 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.120, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.116, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.116 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.172, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.439, 2026-02-25T21:57:53.861 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.476 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.135 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.134, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.205, 2026-02-25T21:57:53.862 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.230 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.130, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.126, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.126 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.141, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.563, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.563 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.156 2026-02-25T21:57:53.863 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.150, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.177, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.209 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.864 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.131, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.130, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.130 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.164, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.307, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.307 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.148 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.865 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.169, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.196, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.185 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.124, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.124, 2026-02-25T21:57:53.866 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.124 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.225, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.392, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.392 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.142 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:47 2026", 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:57:53.867 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.131, 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.180, 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.194 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.107, 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.107, 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.107 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.868 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.170, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.426, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.426 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.098 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.869 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.140, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.171, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.172 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.132, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.122, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.122 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.150, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.379, 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.379 2026-02-25T21:57:53.870 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.158 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.116, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.155, 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.159 2026-02-25T21:57:53.871 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.109, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.109, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.109 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.126, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.302, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.302 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.127 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.872 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:48 2026", 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.117, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.158, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.148 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.102, 2026-02-25T21:57:53.873 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.102, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.102 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.151, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.375, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.375 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.094 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:57:50 2026", 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:57:53.874 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.106, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.153, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.156 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.096, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.096, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.096 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:57:53.875 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.124, 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.302, 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.302 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.124 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout: } 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout: ] 2026-02-25T21:57:53.876 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:57:53.877 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.877 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:57:53.877 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2026-02-25T21:57:53.877 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:57:53.877 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2026-02-25T21:57:53.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:57:53.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2026-02-25T21:58:13.883 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2026-02-25T21:58:13.883 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T21:58:13.884 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T21:58:14.094 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T21:58:14.094 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T21:58:14.094 INFO:tasks.workunit.client.0.trial163.stderr:2' 2026-02-25T21:58:14.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T21:58:14.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:58:14.095 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T21:58:14.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-25T21:58:14.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-25T21:58:14.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836496' 2026-02-25T21:58:14.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:58:14.174 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T21:58:14.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705679 2026-02-25T21:58:14.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705679 2026-02-25T21:58:14.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836496 1-38654705679' 2026-02-25T21:58:14.257 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:58:14.257 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T21:58:14.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-25T21:58:14.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-25T21:58:14.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836496 1-38654705679 2-55834574863' 2026-02-25T21:58:14.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:58:14.335 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836496 2026-02-25T21:58:14.336 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:58:14.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T21:58:14.337 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-25T21:58:14.337 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:58:14.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836496 2026-02-25T21:58:14.339 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836496 2026-02-25T21:58:14.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-25T21:58:14.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:58:14.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-25T21:58:14.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T21:58:15.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T21:58:15.563 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:58:15.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-25T21:58:15.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T21:58:16.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T21:58:16.773 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:58:16.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-25T21:58:16.990 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:58:16.990 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705679 2026-02-25T21:58:16.990 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:58:16.991 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T21:58:16.993 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705679 2026-02-25T21:58:16.993 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:58:16.994 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705679 2026-02-25T21:58:16.994 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 38654705679 2026-02-25T21:58:16.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2026-02-25T21:58:16.995 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T21:58:17.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705680 -lt 38654705679 2026-02-25T21:58:17.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:58:17.216 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574863 2026-02-25T21:58:17.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:58:17.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T21:58:17.218 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-25T21:58:17.219 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:58:17.220 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574863 2026-02-25T21:58:17.220 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-25T21:58:17.220 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 55834574863 2026-02-25T21:58:17.221 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T21:58:17.427 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574863 -lt 55834574863 2026-02-25T21:58:17.428 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:58:17.429 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:58:17.429 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:58:17.429 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:58:17.430 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:58:17.430 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:58:17.430 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:58:17.430 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:58:17.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:58:17.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 2026-02-25T21:58:17.485 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:58:17.485 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 1000, 2026-02-25T21:58:17.485 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:58:17.486 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:58:17.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:58:17.502 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:58:17.503 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.504 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2026-02-25T21:58:17.513 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T21:58:17.514 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:58:17.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:58:17.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:58:17.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:58:17.515 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:58:17.515 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:58:17.516 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:58:17.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:58:17.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:58:17.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 2026-02-25T21:58:17.566 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:58:17.566 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 1000, 2026-02-25T21:58:17.566 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:58:17.566 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:58:17.573 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.573 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:58:17.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:58:17.584 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.584 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2026-02-25T21:58:17.594 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T21:58:17.595 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:58:17.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:58:17.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:58:17.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 0 2026-02-25T21:58:17.651 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 0, 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [ 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:17 2026", 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.182, 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.165, 2026-02-25T21:58:17.652 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.267 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.272, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.272, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 3.982 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.272, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.272, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 3.982 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.172 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.653 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.171, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.182, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.223 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.189, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.283, 2026-02-25T21:58:17.654 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.554 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.189, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.283, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.554 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.187 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:58:17.655 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.164, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.174, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.217 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.210, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.259, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.508 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.210, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.259, 2026-02-25T21:58:17.656 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.508 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.207 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:17 2026", 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.198, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.148, 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.174 2026-02-25T21:58:17.657 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.296, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.296, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.426 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.296, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.296, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.426 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.230 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: } 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout: ] 2026-02-25T21:58:17.658 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:58:17.659 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.660 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:58:17.667 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2026-02-25T21:58:17.668 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.668 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2026-02-25T21:58:17.678 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:58:17.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:58:17.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:58:17.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:58:17.680 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:58:17.680 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:58:17.681 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:58:17.681 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:58:17.681 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:58:17.681 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:58:17.681 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 0 2026-02-25T21:58:17.728 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 0, 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [ 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:12 2026", 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.729 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.144, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.163, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.273 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.126, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.118, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.118 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.174, 2026-02-25T21:58:17.730 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.238, 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 3.982 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.158 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.731 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.171, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.182, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.223 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.155, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.138, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.113 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.189, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.283, 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.554 2026-02-25T21:58:17.732 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.213 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.164, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.174, 2026-02-25T21:58:17.733 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.217 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.134, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.126, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.116 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.210, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.259, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.508 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.184 2026-02-25T21:58:17.734 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.137, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.154, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.201 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.735 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.128, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.128, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.126 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.145, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.193, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.563 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.167 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:15 2026", 2026-02-25T21:58:17.736 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.122, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.145, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.197 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.111, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.111, 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.111 2026-02-25T21:58:17.737 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.145, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.194, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.476 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.126 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:58:17.738 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.131, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.157, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.186 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.117, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.117, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.117 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.739 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.155, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.195, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.307 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.141 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.740 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.138, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.165, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.180 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.101, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.101, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.101 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.159, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.239, 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.392 2026-02-25T21:58:17.741 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.166 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:12 2026", 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.135, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.143, 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.174 2026-02-25T21:58:17.742 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.114, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.098, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.098 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.149, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.323, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.426 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.192 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.743 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.139, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.157, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.167 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.135, 2026-02-25T21:58:17.744 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.120, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.120 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.145, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.222, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.379 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.192 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:58:17.745 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.115, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.127, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.148 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.107, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.097, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.097 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.127, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.177, 2026-02-25T21:58:17.746 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.302 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.152 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:13 2026", 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.092, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.137, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.146 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.747 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.079, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.079, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.079 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.109, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.203, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.375 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.112 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:58:15 2026", 2026-02-25T21:58:17.748 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.103, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.127, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.146 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.087, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.084, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.084 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:58:17.749 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.128, 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.209, 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.302 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.112 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout: } 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout: ] 2026-02-25T21:58:17.750 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:58:17.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:58:17.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2026-02-25T21:58:17.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:58:17.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2026-02-25T21:58:17.758 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:58:17.758 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2026-02-25T21:59:07.760 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2026-02-25T21:59:07.760 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T21:59:07.761 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T21:59:07.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T21:59:07.975 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T21:59:07.975 INFO:tasks.workunit.client.0.trial163.stderr:2' 2026-02-25T21:59:07.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T21:59:07.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:59:07.976 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T21:59:08.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-25T21:59:08.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-25T21:59:08.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509' 2026-02-25T21:59:08.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:59:08.053 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T21:59:08.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705692 2026-02-25T21:59:08.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705692 2026-02-25T21:59:08.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-38654705692' 2026-02-25T21:59:08.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:59:08.130 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T21:59:08.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574875 2026-02-25T21:59:08.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574875 2026-02-25T21:59:08.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574875' 2026-02-25T21:59:08.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:59:08.210 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836509 2026-02-25T21:59:08.210 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:59:08.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T21:59:08.212 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-25T21:59:08.213 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:59:08.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836509 2026-02-25T21:59:08.214 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836509 2026-02-25T21:59:08.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-25T21:59:08.215 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:59:08.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836507 -lt 21474836509 2026-02-25T21:59:08.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T21:59:09.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T21:59:09.442 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:59:09.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836507 -lt 21474836509 2026-02-25T21:59:09.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T21:59:10.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T21:59:10.659 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:59:10.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836509 -lt 21474836509 2026-02-25T21:59:10.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:59:10.870 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705692 2026-02-25T21:59:10.871 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:59:10.872 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T21:59:10.873 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705692 2026-02-25T21:59:10.873 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:59:10.874 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705692 2026-02-25T21:59:10.874 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705692' 2026-02-25T21:59:10.875 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 38654705692 2026-02-25T21:59:10.875 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T21:59:11.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705692 -lt 38654705692 2026-02-25T21:59:11.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:59:11.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574875 2026-02-25T21:59:11.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:59:11.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T21:59:11.090 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574875 2026-02-25T21:59:11.091 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:59:11.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574875 2026-02-25T21:59:11.092 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 55834574875 2026-02-25T21:59:11.093 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574875' 2026-02-25T21:59:11.093 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T21:59:11.311 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574876 -lt 55834574875 2026-02-25T21:59:11.312 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:59:11.312 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:59:11.313 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:59:11.313 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:59:11.314 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:11.314 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:11.314 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:11.314 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:59:11.314 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:59:11.314 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 2026-02-25T21:59:11.368 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:59:11.368 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 1000, 2026-02-25T21:59:11.368 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:59:11.368 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:59:11.375 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.375 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:59:11.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:59:11.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2026-02-25T21:59:11.395 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T21:59:11.395 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:59:11.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:59:11.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:59:11.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:59:11.397 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:11.397 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:11.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:11.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:59:11.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:59:11.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 2026-02-25T21:59:11.447 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:59:11.447 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 1000, 2026-02-25T21:59:11.447 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:59:11.447 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:59:11.454 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.454 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:59:11.463 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:59:11.464 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.464 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2026-02-25T21:59:11.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T21:59:11.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:59:11.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:59:11.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:59:11.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:59:11.477 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:11.477 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:11.477 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:11.477 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:59:11.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:59:11.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 0 2026-02-25T21:59:11.530 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 0, 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [ 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:08 2026", 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.190, 2026-02-25T21:59:11.531 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.177, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.176 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.210, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.211, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.317 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.210, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.211, 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.317 2026-02-25T21:59:11.532 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.169 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:08 2026", 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.176, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.166, 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.176 2026-02-25T21:59:11.533 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.199, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.208, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.330 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.199, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.208, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.330 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.159 2026-02-25T21:59:11.534 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:07 2026", 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.159, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.151, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.154 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.535 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.177, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.177, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.272 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.177, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.177, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.272 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.140 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:07 2026", 2026-02-25T21:59:11.536 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.140, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.136, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.136 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.158, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.158, 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.296 2026-02-25T21:59:11.537 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.158, 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.158, 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.296 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.127 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: } 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout: ] 2026-02-25T21:59:11.538 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:59:11.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:59:11.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2026-02-25T21:59:11.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2026-02-25T21:59:11.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:59:11.559 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:11.560 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:59:11.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:59:11.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 0 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 0, 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [ 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:03 2026", 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:59:11.607 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.176, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.179, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.176 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.166, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.129, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.126 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.181, 2026-02-25T21:59:11.608 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.304, 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.317 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.171 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:08 2026", 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.609 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.177, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.167, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.167 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.131, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.131, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.125 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.217, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.217, 2026-02-25T21:59:11.610 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.232 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.217 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:03 2026", 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.152, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.173, 2026-02-25T21:59:11.611 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.177 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.149, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.129, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.125 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.159, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.313, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.330 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.192 2026-02-25T21:59:11.612 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:03 2026", 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.172, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.171, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.158 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.613 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.162, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.145, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.115 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.183, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.200, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.205 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.166 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:08 2026", 2026-02-25T21:59:11.614 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.162, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.140, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.132 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.127, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.110, 2026-02-25T21:59:11.615 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.093 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.196, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.196, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.196 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.196 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:08 2026", 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:59:11.616 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.162, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.121, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.126 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.149, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.097, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.079 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.182, 2026-02-25T21:59:11.617 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.182, 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.188 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.153 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:08 2026", 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 2, 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.618 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.153, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.149, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.160 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.126, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.120, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.101 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.184, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.188, 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.217 2026-02-25T21:59:11.619 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.184 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:07 2026", 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.620 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.159, 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.151, 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.154 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.143, 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.135, 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.116 2026-02-25T21:59:11.621 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.177, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.177, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.272 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.150 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:03 2026", 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:59:11.622 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 2, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.152, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.155, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.151 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.144, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.113, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.106 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.623 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.160, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.216, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.216 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.187 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:07 2026", 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 0, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 1, 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.624 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.140, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.136, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.136 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.132, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.120, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.090 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.158, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.158, 2026-02-25T21:59:11.625 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.296 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.134 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:05 2026", 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "back", 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.137, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.127, 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.125 2026-02-25T21:59:11.626 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.107, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.098, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.084 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.182, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.182, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.217 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.121 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.627 INFO:tasks.workunit.client.0.trial163.stdout: { 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "last update": "Wed Feb 25 21:59:05 2026", 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "stale": false, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "from osd": 1, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "to osd": 0, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "interface": "front", 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "average": { 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.127, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.135, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.133 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "min": { 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.113, 2026-02-25T21:59:11.628 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.113, 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.099 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: "max": { 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: "1min": 0.135, 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: "5min": 0.165, 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: "15min": 0.194 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: }, 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: "last": 0.146 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: } 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout: ] 2026-02-25T21:59:11.629 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:59:11.630 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.631 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T21:59:11.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2026-02-25T21:59:11.631 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.631 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2026-02-25T21:59:11.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2026-02-25T21:59:11.635 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:59:11.636 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T21:59:11.636 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T21:59:11.636 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T21:59:11.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:11.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:11.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:11.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mgr.x.asok 2026-02-25T21:59:11.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:59:11.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-mgr.x.asok dump_osd_network 99 2026-02-25T21:59:11.683 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:59:11.683 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 99, 2026-02-25T21:59:11.683 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:59:11.683 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:59:11.690 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.690 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2026-02-25T21:59:11.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2026-02-25T21:59:11.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T21:59:11.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T21:59:11.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T21:59:11.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T21:59:11.701 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:11.701 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:11.702 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:11.702 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-osd.0.asok 2026-02-25T21:59:11.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T21:59:11.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.23896/ceph-osd.0.asok dump_osd_network 98 2026-02-25T21:59:11.757 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T21:59:11.757 INFO:tasks.workunit.client.0.trial163.stdout: "threshold": 98, 2026-02-25T21:59:11.757 INFO:tasks.workunit.client.0.trial163.stdout: "entries": [] 2026-02-25T21:59:11.757 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T21:59:11.764 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T21:59:11.764 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2026-02-25T21:59:11.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2026-02-25T21:59:11.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2026-02-25T21:59:11.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-25T21:59:11.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T21:59:11.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:59:11.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T21:59:11.777 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:59:11.777 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:59:11.777 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:59:11.777 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:59:11.777 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:59:12.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:59:12.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:59:12.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:59:12.090 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:59:12.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:59:12.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:59:12.092 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:59:12.094 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:59:12.094 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:59:12.094 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:59:12.094 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:59:12.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:59:12.096 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:59:12.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T21:59:12.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:59:12.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.670 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.23896 2026-02-25T21:59:12.672 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:59:12.672 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T21:59:12.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T21:59:12.674 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T21:59:12.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T21:59:12.675 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T21:59:12.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T21:59:12.676 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T21:59:12.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T21:59:12.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T21:59:12.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T21:59:12.678 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:59:12.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T21:59:12.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T21:59:12.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T21:59:12.680 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T21:59:12.681 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T21:59:12.681 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T21:59:12.682 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T21:59:12.682 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.683 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.683 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.23896 2026-02-25T21:59:12.684 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T21:59:12.684 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T21:59:12.684 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-25T21:59:12.685 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T21:59:12.685 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.685 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.685 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.23896 2026-02-25T21:59:12.687 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T21:59:12.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T21:59:12.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T21:59:12.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T21:59:12.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-25T21:59:12.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-25T21:59:12.712 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T21:59:12.712 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:59:12.712 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:59:12.712 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:59:12.712 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.712 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.713 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:12.713 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T21:59:12.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T21:59:12.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T21:59:12.744 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:59:12.744 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:59:12.744 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.745 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mon.a.asok 2026-02-25T21:59:12.746 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:59:12.746 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.23896/ceph-mon.a.asok config get fsid 2026-02-25T21:59:12.801 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T21:59:12.801 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T21:59:12.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T21:59:12.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.803 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.23896/ceph-mon.a.asok 2026-02-25T21:59:12.804 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T21:59:12.804 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.23896/ceph-mon.a.asok config get mon_host 2026-02-25T21:59:12.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2026-02-25T21:59:12.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-25T21:59:12.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T21:59:12.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T21:59:12.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T21:59:12.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-25T21:59:12.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T21:59:12.972 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T21:59:12.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:59:12.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:59:12.973 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:59:12.974 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.974 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:12.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:59:12.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T21:59:12.997 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2026-02-25T21:59:12.997 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T21:59:12.997 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T21:59:12.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T21:59:12.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T21:59:13.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-25T21:59:13.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:59:13.001 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=416bc285-e038-4952-9d32-4f6ba1fd6f0d 2026-02-25T21:59:13.001 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 416bc285-e038-4952-9d32-4f6ba1fd6f0d' 2026-02-25T21:59:13.001 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 416bc285-e038-4952-9d32-4f6ba1fd6f0d 2026-02-25T21:59:13.002 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:59:13.010 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxcJ9pQQWBABAAPqM9QHkxI8xVbmRWgMsH4g== 2026-02-25T21:59:13.010 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxcJ9pQQWBABAAPqM9QHkxI8xVbmRWgMsH4g=="}' 2026-02-25T21:59:13.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 416bc285-e038-4952-9d32-4f6ba1fd6f0d -i td/network-ping/0/new.json 2026-02-25T21:59:13.125 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:59:13.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-25T21:59:13.134 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxcJ9pQQWBABAAPqM9QHkxI8xVbmRWgMsH4g== --osd-uuid 416bc285-e038-4952-9d32-4f6ba1fd6f0d 2026-02-25T21:59:13.151 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.143+0000 7f40dbed4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:13.152 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.147+0000 7f40dbed4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:13.153 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.147+0000 7f40dbed4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:13.154 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.147+0000 7f40dbed4a40 -1 bdev(0x55ce9063c400 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-25T21:59:13.154 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.147+0000 7f40dbed4a40 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-25T21:59:13.811 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-25T21:59:13.811 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:59:13.812 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T21:59:13.812 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T21:59:13.812 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:59:13.925 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T21:59:13.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T21:59:13.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T21:59:13.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:59:13.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:59:13.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:59:13.942 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.935+0000 7efdf2d24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:13.943 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.939+0000 7efdf2d24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:13.944 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:13.939+0000 7efdf2d24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:14.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T21:59:14.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:59:14.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T21:59:14.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:59:14.040 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:59:14.040 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:14.040 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:59:14.041 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:59:14.041 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:14.041 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:59:14.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:59:14.317 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:14.311+0000 7efdf2d24a40 -1 Falling back to public interface 2026-02-25T21:59:14.942 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:14.935+0000 7efdf2d24a40 -1 osd.0 0 log_to_monitors true 2026-02-25T21:59:15.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:59:15.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:15.151 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:59:15.152 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:59:15.152 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:15.152 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:59:15.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:59:16.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:59:16.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:16.265 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:59:16.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:59:16.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:16.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T21:59:16.471 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/406295375,v1:127.0.0.1:6801/406295375] [v2:127.0.0.1:6802/406295375,v1:127.0.0.1:6803/406295375] exists,up 416bc285-e038-4952-9d32-4f6ba1fd6f0d 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:59:16.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T21:59:16.475 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T21:59:16.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:59:16.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:59:16.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:59:16.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:59:16.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:59:16.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T21:59:16.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-25T21:59:16.478 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:59:16.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=18b5ffcf-dbec-4d78-92ce-1624db929e50 2026-02-25T21:59:16.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 18b5ffcf-dbec-4d78-92ce-1624db929e50' 2026-02-25T21:59:16.478 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 18b5ffcf-dbec-4d78-92ce-1624db929e50 2026-02-25T21:59:16.479 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:59:16.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0cJ9pdpoFHRAANxqt3Zjjjkj8FhlrSmI4vQ== 2026-02-25T21:59:16.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0cJ9pdpoFHRAANxqt3Zjjjkj8FhlrSmI4vQ=="}' 2026-02-25T21:59:16.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 18b5ffcf-dbec-4d78-92ce-1624db929e50 -i td/network-ping/1/new.json 2026-02-25T21:59:16.704 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:59:16.714 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-25T21:59:16.715 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0cJ9pdpoFHRAANxqt3Zjjjkj8FhlrSmI4vQ== --osd-uuid 18b5ffcf-dbec-4d78-92ce-1624db929e50 2026-02-25T21:59:16.732 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:16.727+0000 7fc5a4264a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:16.733 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:16.727+0000 7fc5a4264a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:16.734 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:16.727+0000 7fc5a4264a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:16.735 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:16.727+0000 7fc5a4264a40 -1 bdev(0x5566db880400 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-25T21:59:16.735 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:16.727+0000 7fc5a4264a40 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-25T21:59:17.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-25T21:59:17.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:59:17.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T21:59:17.403 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T21:59:17.404 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:59:17.688 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T21:59:17.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T21:59:17.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T21:59:17.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:59:17.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:59:17.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:59:17.713 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:17.707+0000 7f2998777a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:17.715 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:17.707+0000 7f2998777a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:17.716 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:17.711+0000 7f2998777a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:17.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:59:17.911 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:59:17.911 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:17.911 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T21:59:18.069 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:18.063+0000 7f2998777a40 -1 Falling back to public interface 2026-02-25T21:59:18.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:59:18.662 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:18.655+0000 7f2998777a40 -1 osd.1 0 log_to_monitors true 2026-02-25T21:59:19.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:59:19.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:19.143 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:59:19.144 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:59:19.144 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:19.144 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T21:59:19.364 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/232811464,v1:127.0.0.1:6811/232811464] [v2:127.0.0.1:6812/232811464,v1:127.0.0.1:6813/232811464] exists,up 18b5ffcf-dbec-4d78-92ce-1624db929e50 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T21:59:19.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T21:59:19.368 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T21:59:19.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T21:59:19.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T21:59:19.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T21:59:19.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T21:59:19.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T21:59:19.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T21:59:19.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-25T21:59:19.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T21:59:19.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c5a829a-f633-4e96-a6e4-c7ea55eb0135 2026-02-25T21:59:19.371 INFO:tasks.workunit.client.0.trial163.stdout:add osd2 7c5a829a-f633-4e96-a6e4-c7ea55eb0135 2026-02-25T21:59:19.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7c5a829a-f633-4e96-a6e4-c7ea55eb0135' 2026-02-25T21:59:19.372 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T21:59:19.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC3cJ9p+IefFhAAIhrOClXPYTJqaTgmi6Y0yg== 2026-02-25T21:59:19.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC3cJ9p+IefFhAAIhrOClXPYTJqaTgmi6Y0yg=="}' 2026-02-25T21:59:19.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c5a829a-f633-4e96-a6e4-c7ea55eb0135 -i td/network-ping/2/new.json 2026-02-25T21:59:19.586 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:59:19.595 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-25T21:59:19.596 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3cJ9p+IefFhAAIhrOClXPYTJqaTgmi6Y0yg== --osd-uuid 7c5a829a-f633-4e96-a6e4-c7ea55eb0135 2026-02-25T21:59:19.613 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:19.607+0000 7f9564e6ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:19.614 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:19.607+0000 7f9564e6ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:19.615 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:19.611+0000 7f9564e6ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:19.616 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:19.611+0000 7f9564e6ea40 -1 bdev(0x55a5b2812400 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-25T21:59:19.616 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:19.611+0000 7f9564e6ea40 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-25T21:59:20.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-25T21:59:20.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T21:59:20.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T21:59:20.255 INFO:tasks.workunit.client.0.trial163.stdout:adding osd2 key to auth repository 2026-02-25T21:59:20.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T21:59:20.543 INFO:tasks.workunit.client.0.trial163.stdout:start osd.2 2026-02-25T21:59:20.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T21:59:20.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=2cf6592c-00c5-477e-8966-3ac4183a3239 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.23896/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T21:59:20.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T21:59:20.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T21:59:20.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T21:59:20.560 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:20.555+0000 7f32c1a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:20.562 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:20.555+0000 7f32c1a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:20.562 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:20.555+0000 7f32c1a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:20.755 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T21:59:20.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T21:59:20.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:20.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T21:59:20.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:59:21.017 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:21.011+0000 7f32c1a3fa40 -1 Falling back to public interface 2026-02-25T21:59:21.566 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T21:59:21.559+0000 7f32c1a3fa40 -1 osd.2 0 log_to_monitors true 2026-02-25T21:59:21.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:59:21.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:21.965 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T21:59:21.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T21:59:21.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:21.966 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T21:59:22.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T21:59:23.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T21:59:23.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T21:59:23.189 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T21:59:23.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T21:59:23.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T21:59:23.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T21:59:23.400 INFO:tasks.workunit.client.0.trial163.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3904723551,v1:127.0.0.1:6819/3904723551] [v2:127.0.0.1:6820/3904723551,v1:127.0.0.1:6821/3904723551] exists,up 7c5a829a-f633-4e96-a6e4-c7ea55eb0135 2026-02-25T21:59:23.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T21:59:23.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T21:59:23.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T21:59:23.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2026-02-25T21:59:28.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2026-02-25T21:59:28.656 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'dc1' already exists 2026-02-25T21:59:28.667 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2026-02-25T21:59:28.921 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'dc2' already exists 2026-02-25T21:59:28.931 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2026-02-25T21:59:29.201 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'dc3' already exists 2026-02-25T21:59:29.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2026-02-25T21:59:29.472 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'rack1' already exists 2026-02-25T21:59:29.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2026-02-25T21:59:29.737 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'rack2' already exists 2026-02-25T21:59:29.746 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2026-02-25T21:59:30.011 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'rack3' already exists 2026-02-25T21:59:30.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2026-02-25T21:59:30.291 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'host1' already exists 2026-02-25T21:59:30.302 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2026-02-25T21:59:30.560 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'host2' already exists 2026-02-25T21:59:30.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2026-02-25T21:59:30.822 INFO:tasks.workunit.client.0.trial163.stderr:bucket 'host3' already exists 2026-02-25T21:59:30.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2026-02-25T21:59:31.093 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2026-02-25T21:59:31.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2026-02-25T21:59:31.400 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2026-02-25T21:59:31.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2026-02-25T21:59:31.681 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2026-02-25T21:59:31.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2026-02-25T21:59:31.947 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2026-02-25T21:59:31.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2026-02-25T21:59:32.254 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2026-02-25T21:59:32.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2026-02-25T21:59:32.519 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2026-02-25T21:59:32.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2026-02-25T21:59:32.798 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2026-02-25T21:59:32.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2026-02-25T21:59:33.074 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2026-02-25T21:59:33.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2026-02-25T21:59:33.352 INFO:tasks.workunit.client.0.trial163.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2026-02-25T21:59:33.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2026-02-25T21:59:33.631 INFO:tasks.workunit.client.0.trial163.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2026-02-25T21:59:33.641 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2026-02-25T21:59:33.911 INFO:tasks.workunit.client.0.trial163.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2026-02-25T21:59:33.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2026-02-25T21:59:34.215 INFO:tasks.workunit.client.0.trial163.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2026-02-25T21:59:34.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2026-02-25T21:59:34.494 INFO:tasks.workunit.client.0.trial163.stderr:rule myrule already exists 2026-02-25T21:59:34.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2026-02-25T21:59:34.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 16 replicated myrule 2026-02-25T21:59:34.774 INFO:tasks.workunit.client.0.trial163.stderr:pool 'foo' already exists 2026-02-25T21:59:34.784 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T21:59:35.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-25T21:59:35.807 INFO:tasks.workunit.client.0.trial163.stdout:hints = 1 2026-02-25T21:59:35.807 INFO:tasks.workunit.client.0.trial163.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-25T21:59:35.807 INFO:tasks.workunit.client.0.trial163.stdout:Object prefix: benchmark_data_trial163_35665 2026-02-25T21:59:35.807 INFO:tasks.workunit.client.0.trial163.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-25T21:59:35.807 INFO:tasks.workunit.client.0.trial163.stdout: 0 0 0 0 0 0 - 0 2026-02-25T21:59:36.807 INFO:tasks.workunit.client.0.trial163.stdout: 1 16 10631 10615 41.4616 41.4648 0.00136389 0.0015049 2026-02-25T21:59:37.807 INFO:tasks.workunit.client.0.trial163.stdout: 2 16 21641 21625 42.2331 43.0078 0.000626642 0.00147795 2026-02-25T21:59:38.807 INFO:tasks.workunit.client.0.trial163.stdout: 3 16 32742 32726 42.6087 43.3633 0.001526 0.00146504 2026-02-25T21:59:39.807 INFO:tasks.workunit.client.0.trial163.stdout: 4 16 43857 43841 42.8098 43.418 0.00133077 0.00145832 2026-02-25T21:59:40.807 INFO:tasks.workunit.client.0.trial163.stdout: 5 16 54775 54759 42.7769 42.6484 0.00090017 0.00145952 2026-02-25T21:59:41.807 INFO:tasks.workunit.client.0.trial163.stdout: 6 16 66025 66009 42.9711 43.9453 0.000984957 0.00145295 2026-02-25T21:59:42.807 INFO:tasks.workunit.client.0.trial163.stdout: 7 15 77415 77400 43.1885 44.4961 0.00106373 0.00144565 2026-02-25T21:59:43.807 INFO:tasks.workunit.client.0.trial163.stdout: 8 16 87826 87810 42.8725 40.6641 0.00111711 0.00145627 2026-02-25T21:59:44.807 INFO:tasks.workunit.client.0.trial163.stdout: 9 16 97399 97383 42.2637 37.3945 0.00132927 0.00147732 2026-02-25T21:59:45.807 INFO:tasks.workunit.client.0.trial163.stdout: 10 8 106946 106938 41.7694 37.3242 0.00170428 0.00149483 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Total time run: 10.0011 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Total writes made: 106946 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Write size: 4096 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Object size: 4096 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Bandwidth (MB/sec): 41.771 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Stddev Bandwidth: 2.58086 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Max bandwidth (MB/sec): 44.4961 2026-02-25T21:59:46.808 INFO:tasks.workunit.client.0.trial163.stdout:Min bandwidth (MB/sec): 37.3242 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Average IOPS: 10693 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Stddev IOPS: 660.7 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Max IOPS: 11391 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Min IOPS: 9555 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Average Latency(s): 0.00149481 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Stddev Latency(s): 0.000739569 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Max latency(s): 0.01094 2026-02-25T21:59:46.809 INFO:tasks.workunit.client.0.trial163.stdout:Min latency(s): 0.000475683 2026-02-25T21:59:46.814 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2026-02-25T21:59:56.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2026-02-25T21:59:56.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T21:59:56.816 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T21:59:57.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T21:59:57.024 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T21:59:57.024 INFO:tasks.workunit.client.0.trial163.stderr:2' 2026-02-25T21:59:57.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T21:59:57.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:59:57.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T21:59:57.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-25T21:59:57.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-25T21:59:57.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491' 2026-02-25T21:59:57.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:59:57.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T21:59:57.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-25T21:59:57.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-25T21:59:57.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2026-02-25T21:59:57.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T21:59:57.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T21:59:57.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-25T21:59:57.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-25T21:59:57.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542153' 2026-02-25T21:59:57.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:59:57.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836491 2026-02-25T21:59:57.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:59:57.266 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T21:59:57.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-25T21:59:57.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:59:57.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836491 2026-02-25T21:59:57.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-25T21:59:57.269 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836491 2026-02-25T21:59:57.269 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T21:59:57.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-25T21:59:57.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:59:57.482 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705674 2026-02-25T21:59:57.482 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:59:57.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T21:59:57.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-25T21:59:57.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:59:57.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705674 2026-02-25T21:59:57.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-25T21:59:57.486 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 38654705674 2026-02-25T21:59:57.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T21:59:57.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-25T21:59:57.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T21:59:57.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542153 2026-02-25T21:59:57.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T21:59:57.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T21:59:57.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-25T21:59:57.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T21:59:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542153 2026-02-25T21:59:57.702 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 60129542153 2026-02-25T21:59:57.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-25T21:59:57.702 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T21:59:57.929 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542151 -lt 60129542153 2026-02-25T21:59:57.929 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T21:59:58.930 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T21:59:58.930 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T21:59:59.136 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542151 -lt 60129542153 2026-02-25T21:59:59.136 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:00.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:00:00.138 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:00:00.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-25T22:00:00.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2026-02-25T22:00:00.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2026-02-25T22:00:00.596 INFO:tasks.workunit.client.0.trial163.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.391ms); Slow OSD heartbeats on front (longest 0.425ms) 2026-02-25T22:00:00.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2026-02-25T22:00:00.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2026-02-25T22:00:00.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2026-02-25T22:00:00.881 INFO:tasks.workunit.client.0.trial163.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.391ms); Slow OSD heartbeats on front (longest 0.425ms) 2026-02-25T22:00:00.881 INFO:tasks.workunit.client.0.trial163.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.391ms) 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.391 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.215 msec possibly improving 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.211 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.182 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.170 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.164 msec possibly improving 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.425ms) 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.425 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.381 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.277 msec 2026-02-25T22:00:00.882 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.223 msec 2026-02-25T22:00:00.883 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.215 msec 2026-02-25T22:00:00.883 INFO:tasks.workunit.client.0.trial163.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.196 msec possibly improving 2026-02-25T22:00:00.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2026-02-25T22:00:00.892 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2026-02-25T22:00:00.892 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2026-02-25T22:00:00.894 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2026-02-25T22:00:00.895 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-25T22:00:00.895 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:00:00.895 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:00.896 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:00:00.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:00.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:00.897 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:00.897 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:00.897 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:01.207 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:01.207 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:01.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:01.209 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:01.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:01.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:01.212 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:01.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:01.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:01.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:01.214 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:01.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:01.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:01.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:00:01.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:01.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:01.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T22:00:01.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.23896 2026-02-25T22:00:01.924 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:01.924 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:01.924 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/network-ping 0 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:01.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:01.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:01.927 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:01.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:01.928 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:01.929 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:01.929 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:01.930 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:01.931 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:01.931 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:01.931 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:01.931 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:01.932 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:01.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:00:01.934 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:00:01.935 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:01.935 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:01.936 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.23896 2026-02-25T22:00:01.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.23896 2026-02-25T22:00:01.937 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:01.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:01.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:00:01.938 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:00:01.939 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:00:01.987 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2026-02-25T22:00:01.987 DEBUG:teuthology.orchestra.run.trial163:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2026-02-25T22:00:02.040 INFO:tasks.workunit.client.0.trial163.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:00:02.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/ok-to-stop 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2026-02-25T22:00:02.045 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2026-02-25T22:00:02.046 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2026-02-25T22:00:02.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2026-02-25T22:00:02.046 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none ' 2026-02-25T22:00:02.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none ' 2026-02-25T22:00:02.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none ' 2026-02-25T22:00:02.047 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2026-02-25T22:00:02.047 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:00:02.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:TEST_0_osd 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:TEST_1_mon_checks 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:TEST_2_mons_checks 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:TEST_3_mons_checks 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:TEST_4_mons_checks 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:TEST_5_mons_checks' 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:02.049 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:02.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:02.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:02.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:02.052 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:02.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:02.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:02.053 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:02.054 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:02.054 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:02.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:02.055 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:02.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:02.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:02.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:02.059 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:02.059 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.059 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.059 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:02.060 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:02.060 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:02.060 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:00:02.061 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:00:02.062 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.062 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.062 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:00:02.063 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:02.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:00:02.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:02.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:00:02.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:00:02.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:02.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:02.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:02.088 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:02.089 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.089 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:02.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:00:02.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:02.115 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:00:02.115 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:02.115 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:02.115 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:02.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:00:02.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:00:02.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:02.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:02.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:02.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:02.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:00:02.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2026-02-25T22:00:02.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-25T22:00:02.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:00:02.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:00:02.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:00:02.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-25T22:00:02.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:00:02.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:00:02.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:02.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:02.332 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:02.333 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.333 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.333 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:02.333 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:00:02.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-25T22:00:02.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:02.355 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:00:02.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:00:02.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:00:02.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:00:02.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:00:02.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-25T22:00:02.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:00:02.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d9ebc82e-0875-47cc-8bcf-fd65e27f05d2 2026-02-25T22:00:02.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d9ebc82e-0875-47cc-8bcf-fd65e27f05d2' 2026-02-25T22:00:02.358 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 d9ebc82e-0875-47cc-8bcf-fd65e27f05d2 2026-02-25T22:00:02.358 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:00:02.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDicJ9pynvKFRAAtmA/nuj9gdoTZv2oqYAYSg== 2026-02-25T22:00:02.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDicJ9pynvKFRAAtmA/nuj9gdoTZv2oqYAYSg=="}' 2026-02-25T22:00:02.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d9ebc82e-0875-47cc-8bcf-fd65e27f05d2 -i td/ok-to-stop/0/new.json 2026-02-25T22:00:02.480 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:02.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-25T22:00:02.490 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDicJ9pynvKFRAAtmA/nuj9gdoTZv2oqYAYSg== --osd-uuid d9ebc82e-0875-47cc-8bcf-fd65e27f05d2 2026-02-25T22:00:02.504 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:02.498+0000 7f4d370f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:02.506 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:02.498+0000 7f4d370f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:02.507 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:02.502+0000 7f4d370f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:02.508 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:02.502+0000 7f4d370f9a40 -1 bdev(0x55da13f44400 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:00:02.508 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:02.502+0000 7f4d370f9a40 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-25T22:00:03.237 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-25T22:00:03.237 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:00:03.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:00:03.238 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:00:03.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:00:03.355 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:00:03.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:00:03.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:00:03.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:00:03.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:00:03.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:00:03.377 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:03.370+0000 7fe9ee44fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:03.379 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:03.374+0000 7fe9ee44fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:03.380 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:03.374+0000 7fe9ee44fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:03.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:00:03.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:00:03.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:00:03.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:00:03.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:00:03.463 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:03.463 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:03.463 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:00:03.464 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:03.464 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:00:03.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:03.760 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:03.754+0000 7fe9ee44fa40 -1 Falling back to public interface 2026-02-25T22:00:04.387 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:04.382+0000 7fe9ee44fa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:00:04.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:04.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:04.571 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:00:04.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:00:04.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:04.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:00:04.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:05.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:05.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:05.702 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:00:05.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:00:05.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:05.703 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:00:05.919 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1972510557,v1:127.0.0.1:6801/1972510557] [v2:127.0.0.1:6802/1972510557,v1:127.0.0.1:6803/1972510557] exists,up d9ebc82e-0875-47cc-8bcf-fd65e27f05d2 2026-02-25T22:00:05.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:00:05.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:00:05.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:00:05.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2026-02-25T22:00:05.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=a 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/a 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:05.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:05.923 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:05.945 INFO:tasks.workunit.client.0.trial163.stdout:starting mds.a at 2026-02-25T22:00:05.949 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2026-02-25T22:00:06.227 INFO:tasks.workunit.client.0.trial163.stderr:pool 'meta' already exists 2026-02-25T22:00:06.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2026-02-25T22:00:06.531 INFO:tasks.workunit.client.0.trial163.stderr:pool 'data' already exists 2026-02-25T22:00:06.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2026-02-25T22:00:06.885 INFO:tasks.workunit.client.0.trial163.stderr:filesystem 'myfs' already exists 2026-02-25T22:00:06.897 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2026-02-25T22:00:11.898 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2026-02-25T22:00:12.083 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 2026-02-25T22:00:12.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2026-02-25T22:00:12.272 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 2026-02-25T22:00:12.277 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2026-02-25T22:00:12.543 INFO:tasks.workunit.client.0.trial163.stderr:should be safe to stop [dne] 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=b 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/b 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:12.554 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:12.555 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:12.555 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:12.555 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:12.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:12.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:12.569 INFO:tasks.workunit.client.0.trial163.stdout:starting mds.b at 2026-02-25T22:00:12.573 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2026-02-25T22:00:17.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2026-02-25T22:00:17.843 INFO:tasks.workunit.client.0.trial163.stderr:should be safe to stop [a] 2026-02-25T22:00:17.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2026-02-25T22:00:18.143 INFO:tasks.workunit.client.0.trial163.stderr:should be safe to stop [b] 2026-02-25T22:00:18.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2026-02-25T22:00:18.340 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 4137 2026-02-25T22:00:18.345 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2026-02-25T22:00:18.614 INFO:tasks.workunit.client.0.trial163.stderr:should be safe to stop [a,dne1,dne2] 2026-02-25T22:00:18.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2026-02-25T22:00:18.907 INFO:tasks.workunit.client.0.trial163.stderr:should be safe to stop [b,dne] 2026-02-25T22:00:18.916 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2026-02-25T22:00:19.120 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 4137 2026-02-25T22:00:19.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2026-02-25T22:00:19.390 INFO:tasks.workunit.client.0.trial163.stderr:should be safe to stop [dne1,dne2] 2026-02-25T22:00:19.400 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2026-02-25T22:00:19.400 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:19.400 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:19.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:19.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:19.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:19.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:19.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:19.508 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:19.509 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:19.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:19.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:19.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:19.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:19.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:19.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:19.624 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:19.625 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:19.625 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:19.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:19.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:19.627 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:19.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:19.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:19.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:19.629 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:19.630 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:19.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:19.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:19.635 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:19.636 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.636 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.636 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:19.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:19.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:19.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:00:19.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:19.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:19.640 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:19.641 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:19.641 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:19.642 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:19.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:19.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:19.643 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:19.644 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:19.645 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:19.645 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:19.645 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:19.646 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:19.647 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:19.647 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:19.649 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:19.649 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.649 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:19.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:19.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:19.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:00:19.652 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:00:19.652 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.652 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:00:19.653 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:00:19.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:19.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:19.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:00:19.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:19.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:00:19.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:00:19.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:19.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:19.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:19.677 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:19.677 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.677 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.678 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:19.678 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:00:19.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:19.705 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:00:19.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:19.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:19.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:19.707 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:19.707 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:19.707 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:19.707 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:19.707 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:19.708 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.708 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.708 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:19.708 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:19.708 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:00:19.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:00:19.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:19.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:19.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:19.763 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:19.764 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:00:19.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2026-02-25T22:00:19.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-25T22:00:19.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:00:19.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:00:19.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:00:19.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-25T22:00:19.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:00:19.924 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:00:19.924 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:19.924 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:19.924 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:19.924 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.924 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.925 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:19.925 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:00:19.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:00:19.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:00:19.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:19.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-25T22:00:19.948 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:00:19.949 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f36e0323-fc7b-46b8-beb3-09820ac186d4 2026-02-25T22:00:19.949 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 f36e0323-fc7b-46b8-beb3-09820ac186d4 2026-02-25T22:00:19.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f36e0323-fc7b-46b8-beb3-09820ac186d4' 2026-02-25T22:00:19.950 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:00:19.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDzcJ9ppUQUORAA5fnvIxxV6k7yFokOCRht8A== 2026-02-25T22:00:19.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDzcJ9ppUQUORAA5fnvIxxV6k7yFokOCRht8A=="}' 2026-02-25T22:00:19.959 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f36e0323-fc7b-46b8-beb3-09820ac186d4 -i td/ok-to-stop/0/new.json 2026-02-25T22:00:20.070 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:20.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-25T22:00:20.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzcJ9ppUQUORAA5fnvIxxV6k7yFokOCRht8A== --osd-uuid f36e0323-fc7b-46b8-beb3-09820ac186d4 2026-02-25T22:00:20.095 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:20.091+0000 7f13188d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:20.097 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:20.091+0000 7f13188d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:20.098 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:20.091+0000 7f13188d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:20.098 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:20.095+0000 7f13188d7a40 -1 bdev(0x55c6f3730400 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:00:20.098 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:20.095+0000 7f13188d7a40 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-25T22:00:20.874 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-25T22:00:20.874 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:00:20.875 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:00:20.875 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:00:20.876 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:00:20.993 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:00:20.994 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:00:20.994 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:00:20.994 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:00:20.994 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:00:20.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:00:21.017 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:21.011+0000 7fbdf5b65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:21.019 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:21.015+0000 7fbdf5b65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:21.020 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:21.015+0000 7fbdf5b65a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:21.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:00:21.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:00:21.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:00:21.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:00:21.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:00:21.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:21.102 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:21.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:00:21.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:21.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:00:21.206 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:21.435 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:21.431+0000 7fbdf5b65a40 -1 Falling back to public interface 2026-02-25T22:00:22.063 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:22.059+0000 7fbdf5b65a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:00:22.208 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:22.208 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:22.208 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:00:22.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:00:22.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:22.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:00:22.321 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:23.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:23.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:23.323 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:00:23.324 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:00:23.324 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:23.324 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:00:23.533 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1054390997,v1:127.0.0.1:6801/1054390997] [v2:127.0.0.1:6802/1054390997,v1:127.0.0.1:6803/1054390997] exists,up f36e0323-fc7b-46b8-beb3-09820ac186d4 2026-02-25T22:00:23.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:00:23.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:00:23.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/1 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:00:23.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:23.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/1 2026-02-25T22:00:23.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:00:23.540 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=96e79c70-c28c-4373-8df7-7683b5d8ac6a 2026-02-25T22:00:23.540 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 96e79c70-c28c-4373-8df7-7683b5d8ac6a 2026-02-25T22:00:23.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 96e79c70-c28c-4373-8df7-7683b5d8ac6a' 2026-02-25T22:00:23.541 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:00:23.549 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD3cJ9p7uqwIBAA4BhyM3EqDzxsrJeMWeq/Qg== 2026-02-25T22:00:23.549 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD3cJ9p7uqwIBAA4BhyM3EqDzxsrJeMWeq/Qg=="}' 2026-02-25T22:00:23.549 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 96e79c70-c28c-4373-8df7-7683b5d8ac6a -i td/ok-to-stop/1/new.json 2026-02-25T22:00:23.759 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:00:23.768 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/1/new.json 2026-02-25T22:00:23.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3cJ9p7uqwIBAA4BhyM3EqDzxsrJeMWeq/Qg== --osd-uuid 96e79c70-c28c-4373-8df7-7683b5d8ac6a 2026-02-25T22:00:23.785 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:23.779+0000 7f2bc037ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:23.787 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:23.783+0000 7f2bc037ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:23.788 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:23.783+0000 7f2bc037ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:23.788 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:23.783+0000 7f2bc037ea40 -1 bdev(0x558da3b28400 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:00:23.788 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:23.783+0000 7f2bc037ea40 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2026-02-25T22:00:24.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/1/keyring 2026-02-25T22:00:24.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:00:24.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:00:24.462 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T22:00:24.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:00:24.754 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T22:00:24.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:00:24.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:00:24.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:00:24.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:00:24.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:00:24.768 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:24.763+0000 7fedcb73ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:24.770 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:24.763+0000 7fedcb73ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:24.771 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:24.767+0000 7fedcb73ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:24.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:00:24.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:00:24.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:00:24.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:00:24.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:00:24.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:24.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:00:24.964 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:24.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:24.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:00:25.143 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:25.139+0000 7fedcb73ca40 -1 Falling back to public interface 2026-02-25T22:00:25.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:25.764 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:25.759+0000 7fedcb73ca40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:00:26.173 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:26.173 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:26.173 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:00:26.173 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:00:26.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:26.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:00:26.386 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:27.388 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:27.388 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:27.388 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:00:27.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:00:27.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:27.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:00:27.597 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4233430424,v1:127.0.0.1:6811/4233430424] [v2:127.0.0.1:6812/4233430424,v1:127.0.0.1:6813/4233430424] exists,up 96e79c70-c28c-4373-8df7-7683b5d8ac6a 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:00:27.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/2 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:00:27.602 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:00:27.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/2 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:00:27.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=266c0c2c-e252-4182-84e7-327a70cff153 2026-02-25T22:00:27.605 INFO:tasks.workunit.client.0.trial163.stdout:add osd2 266c0c2c-e252-4182-84e7-327a70cff153 2026-02-25T22:00:27.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 266c0c2c-e252-4182-84e7-327a70cff153' 2026-02-25T22:00:27.605 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:00:27.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD7cJ9pM6SSJBAAW27MiKJISsJKDVXcjicZGw== 2026-02-25T22:00:27.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD7cJ9pM6SSJBAAW27MiKJISsJKDVXcjicZGw=="}' 2026-02-25T22:00:27.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 266c0c2c-e252-4182-84e7-327a70cff153 -i td/ok-to-stop/2/new.json 2026-02-25T22:00:27.827 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:00:27.837 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/2/new.json 2026-02-25T22:00:27.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7cJ9pM6SSJBAAW27MiKJISsJKDVXcjicZGw== --osd-uuid 266c0c2c-e252-4182-84e7-327a70cff153 2026-02-25T22:00:27.854 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:27.847+0000 7f0e13e3aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:27.855 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:27.851+0000 7f0e13e3aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:27.856 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:27.851+0000 7f0e13e3aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:27.857 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:27.851+0000 7f0e13e3aa40 -1 bdev(0x55704fde0400 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:00:27.857 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:27.851+0000 7f0e13e3aa40 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2026-02-25T22:00:28.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/2/keyring 2026-02-25T22:00:28.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:00:28.458 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:00:28.459 INFO:tasks.workunit.client.0.trial163.stdout:adding osd2 key to auth repository 2026-02-25T22:00:28.459 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:00:28.754 INFO:tasks.workunit.client.0.trial163.stdout:start osd.2 2026-02-25T22:00:28.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:00:28.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:00:28.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:00:28.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:00:28.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:00:28.776 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:28.771+0000 7f06ad2dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:28.778 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:28.775+0000 7f06ad2dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:28.779 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:28.775+0000 7f06ad2dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:28.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:00:28.966 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:28.966 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:28.966 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:00:29.115 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:29.111+0000 7f06ad2dba40 -1 Falling back to public interface 2026-02-25T22:00:29.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:29.745 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:29.739+0000 7f06ad2dba40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:00:30.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:30.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:30.182 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:00:30.183 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:00:30.183 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:30.183 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:00:30.386 INFO:tasks.workunit.client.0.trial163.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/451576843,v1:127.0.0.1:6819/451576843] [v2:127.0.0.1:6820/451576843,v1:127.0.0.1:6821/451576843] exists,up 266c0c2c-e252-4182-84e7-327a70cff153 2026-02-25T22:00:30.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:00:30.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:00:30.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:00:30.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2026-02-25T22:00:30.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:00:30.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/3 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:00:30.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:00:30.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/3 2026-02-25T22:00:30.393 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:00:30.394 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3af54629-0f2f-402b-9723-49eb00efa01f 2026-02-25T22:00:30.394 INFO:tasks.workunit.client.0.trial163.stdout:add osd3 3af54629-0f2f-402b-9723-49eb00efa01f 2026-02-25T22:00:30.394 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 3af54629-0f2f-402b-9723-49eb00efa01f' 2026-02-25T22:00:30.395 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:00:30.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+cJ9pUQIFGBAAQvkUDJOuVN4mSXAggtSfrQ== 2026-02-25T22:00:30.404 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+cJ9pUQIFGBAAQvkUDJOuVN4mSXAggtSfrQ=="}' 2026-02-25T22:00:30.404 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3af54629-0f2f-402b-9723-49eb00efa01f -i td/ok-to-stop/3/new.json 2026-02-25T22:00:30.621 INFO:tasks.workunit.client.0.trial163.stdout:3 2026-02-25T22:00:30.630 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/3/new.json 2026-02-25T22:00:30.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+cJ9pUQIFGBAAQvkUDJOuVN4mSXAggtSfrQ== --osd-uuid 3af54629-0f2f-402b-9723-49eb00efa01f 2026-02-25T22:00:30.646 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:30.643+0000 7f615b7c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:30.647 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:30.643+0000 7f615b7c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:30.648 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:30.643+0000 7f615b7c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:30.649 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:30.643+0000 7f615b7c9a40 -1 bdev(0x555e35298400 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:00:30.649 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:30.643+0000 7f615b7c9a40 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2026-02-25T22:00:31.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/3/keyring 2026-02-25T22:00:31.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:00:31.373 INFO:tasks.workunit.client.0.trial163.stdout:adding osd3 key to auth repository 2026-02-25T22:00:31.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:00:31.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:00:31.655 INFO:tasks.workunit.client.0.trial163.stdout:start osd.3 2026-02-25T22:00:31.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:00:31.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:00:31.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:00:31.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:00:31.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:00:31.677 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:31.671+0000 7f5b48045a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:31.679 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:31.675+0000 7f5b48045a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:31.680 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:31.675+0000 7f5b48045a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:00:31.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:00:31.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:00:31.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:00:31.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:00:31.872 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:00:31.872 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:31.872 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:00:31.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:00:31.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:31.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:00:32.047 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:32.043+0000 7f5b48045a40 -1 Falling back to public interface 2026-02-25T22:00:32.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:00:32.659 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:00:32.655+0000 7f5b48045a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:00:33.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:00:33.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:00:33.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:00:33.091 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:00:33.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:00:33.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:00:33.300 INFO:tasks.workunit.client.0.trial163.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/617182224,v1:127.0.0.1:6827/617182224] [v2:127.0.0.1:6828/617182224,v1:127.0.0.1:6829/617182224] exists,up 3af54629-0f2f-402b-9723-49eb00efa01f 2026-02-25T22:00:33.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:00:33.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:00:33.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:00:33.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2026-02-25T22:00:33.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2026-02-25T22:00:33.928 INFO:tasks.workunit.client.0.trial163.stderr:pool 'ec' already exists 2026-02-25T22:00:33.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2026-02-25T22:00:33.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:00:33.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:00:33.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:00:33.939 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:00:33.940 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:00:33.940 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:00:33.940 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:00:33.940 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:00:33.940 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:00:34.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:00:34.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:00:34.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:00:34.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:00:34.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:00:34.048 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr:2 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr:3' 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:34.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:00:34.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:00:34.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:00:34.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:00:34.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:34.334 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:00:34.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-25T22:00:34.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-25T22:00:34.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-25T22:00:34.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:34.413 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:00:34.491 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-25T22:00:34.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-25T22:00:34.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-60129542147' 2026-02-25T22:00:34.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:34.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:00:34.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411331 2026-02-25T22:00:34.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411331 2026-02-25T22:00:34.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-60129542147 3-77309411331' 2026-02-25T22:00:34.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:34.568 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:00:34.568 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:34.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:00:34.570 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:00:34.570 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:34.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:00:34.571 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:00:34.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:00:34.572 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:34.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-25T22:00:34.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:35.794 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:00:35.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:36.001 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-25T22:00:36.001 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:37.002 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:00:37.003 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:37.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-25T22:00:37.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:37.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672964 2026-02-25T22:00:37.218 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:37.219 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:00:37.219 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-25T22:00:37.219 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:37.221 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672964 2026-02-25T22:00:37.221 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-25T22:00:37.221 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 42949672964 2026-02-25T22:00:37.221 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:00:37.437 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-25T22:00:37.437 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:37.438 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542147 2026-02-25T22:00:37.438 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:37.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:00:37.439 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-25T22:00:37.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:37.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542147 2026-02-25T22:00:37.441 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 60129542147 2026-02-25T22:00:37.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-25T22:00:37.442 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:00:37.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542148 -lt 60129542147 2026-02-25T22:00:37.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:37.659 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-77309411331 2026-02-25T22:00:37.659 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:37.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:00:37.660 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411331 2026-02-25T22:00:37.661 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:37.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411331 2026-02-25T22:00:37.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 77309411331' 2026-02-25T22:00:37.662 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.3 seq 77309411331 2026-02-25T22:00:37.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:00:37.877 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411331 -lt 77309411331 2026-02-25T22:00:37.878 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:00:37.878 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:37.879 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:38.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-25T22:00:38.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:00:38.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:00:38.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:00:38.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:00:38.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:00:38.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:00:38.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:00:38.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-25T22:00:38.384 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:00:38.384 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:38.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:38.671 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-25T22:00:38.671 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:00:38.671 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:00:38.671 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2026-02-25T22:00:38.925 INFO:tasks.workunit.client.0.trial163.stderr:set pool 1 min_size to 3 2026-02-25T22:00:38.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2026-02-25T22:00:38.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:00:38.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:00:38.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:00:38.939 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:00:38.939 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:00:38.939 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:00:38.940 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:00:38.940 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:00:38.940 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:00:39.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:00:39.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:00:39.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:00:39.050 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:00:39.051 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:00:39.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:00:39.258 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:00:39.258 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T22:00:39.258 INFO:tasks.workunit.client.0.trial163.stderr:2 2026-02-25T22:00:39.258 INFO:tasks.workunit.client.0.trial163.stderr:3' 2026-02-25T22:00:39.258 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:00:39.258 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:39.259 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:00:39.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-25T22:00:39.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-25T22:00:39.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488' 2026-02-25T22:00:39.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:39.338 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:00:39.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-25T22:00:39.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-25T22:00:39.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2026-02-25T22:00:39.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:39.414 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:00:39.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-25T22:00:39.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-25T22:00:39.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-60129542150' 2026-02-25T22:00:39.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:39.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:00:39.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-25T22:00:39.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-25T22:00:39.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-60129542150 3-77309411334' 2026-02-25T22:00:39.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:39.572 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836488 2026-02-25T22:00:39.572 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:39.573 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:00:39.574 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-25T22:00:39.574 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:39.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836488 2026-02-25T22:00:39.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-25T22:00:39.576 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836488 2026-02-25T22:00:39.576 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:39.791 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836488 2026-02-25T22:00:39.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:40.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:00:40.793 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:41.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-25T22:00:41.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:41.005 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672967 2026-02-25T22:00:41.005 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:41.006 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:00:41.007 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-25T22:00:41.007 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:41.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672967 2026-02-25T22:00:41.008 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 42949672967 2026-02-25T22:00:41.009 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-25T22:00:41.009 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:00:41.222 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672967 -lt 42949672967 2026-02-25T22:00:41.222 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:41.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542150 2026-02-25T22:00:41.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:41.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:00:41.225 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-25T22:00:41.225 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:41.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542150 2026-02-25T22:00:41.226 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 60129542150 2026-02-25T22:00:41.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-25T22:00:41.227 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:00:41.446 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-25T22:00:41.446 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:41.447 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-77309411334 2026-02-25T22:00:41.447 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:41.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:00:41.449 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411334 2026-02-25T22:00:41.449 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:41.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411334 2026-02-25T22:00:41.450 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.3 seq 77309411334 2026-02-25T22:00:41.451 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 77309411334' 2026-02-25T22:00:41.451 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:00:41.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411334 -lt 77309411334 2026-02-25T22:00:41.664 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:00:41.664 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:41.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:41.951 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-25T22:00:41.951 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:00:41.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:00:41.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:00:41.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:00:41.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:00:41.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:00:41.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:00:42.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-25T22:00:42.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:00:42.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:42.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:42.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-25T22:00:42.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:00:42.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:00:42.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-25T22:00:42.655 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:42.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2026-02-25T22:00:42.873 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:42.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2026-02-25T22:00:43.095 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:43.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2026-02-25T22:00:43.317 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:43.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-25T22:00:43.488 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:43.489 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:00:43.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-25T22:00:43.637 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:43.638 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:00:43.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-25T22:00:43.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2026-02-25T22:00:43.858 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:43.859 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2026-02-25T22:00:43.859 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2026-02-25T22:00:44.069 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:44.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2026-02-25T22:00:44.370 INFO:tasks.workunit.client.0.trial163.stderr:set pool 1 min_size to 2 2026-02-25T22:00:44.385 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2026-02-25T22:00:44.385 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:00:44.386 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:00:44.386 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:00:44.386 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:00:44.387 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:00:44.387 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:00:44.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:00:44.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:00:44.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:00:44.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:00:44.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:00:44.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:00:44.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:00:44.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:00:44.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:00:44.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:00:44.699 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T22:00:44.699 INFO:tasks.workunit.client.0.trial163.stderr:2 2026-02-25T22:00:44.699 INFO:tasks.workunit.client.0.trial163.stderr:3' 2026-02-25T22:00:44.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:00:44.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:44.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:00:44.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-25T22:00:44.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-25T22:00:44.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491' 2026-02-25T22:00:44.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:44.779 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:00:44.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-25T22:00:44.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-25T22:00:44.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-25T22:00:44.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:44.855 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:00:44.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-25T22:00:44.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-25T22:00:44.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153' 2026-02-25T22:00:44.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:44.936 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:00:45.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-25T22:00:45.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-25T22:00:45.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-77309411337' 2026-02-25T22:00:45.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:45.015 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836491 2026-02-25T22:00:45.015 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:45.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:00:45.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-25T22:00:45.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:45.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836491 2026-02-25T22:00:45.018 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836491 2026-02-25T22:00:45.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-25T22:00:45.018 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:45.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-25T22:00:45.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:46.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:00:46.227 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:00:46.444 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-25T22:00:46.444 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:46.445 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672970 2026-02-25T22:00:46.445 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:46.446 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:00:46.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-25T22:00:46.447 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:46.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672970 2026-02-25T22:00:46.448 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 42949672970 2026-02-25T22:00:46.449 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-25T22:00:46.449 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:00:46.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-25T22:00:46.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:46.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542153 2026-02-25T22:00:46.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:46.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:00:46.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-25T22:00:46.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:46.666 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542153 2026-02-25T22:00:46.667 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 60129542153 2026-02-25T22:00:46.667 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-25T22:00:46.667 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:00:46.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-25T22:00:46.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:46.881 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-77309411337 2026-02-25T22:00:46.881 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:46.883 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:00:46.883 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411337 2026-02-25T22:00:46.884 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:46.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411337 2026-02-25T22:00:46.885 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.3 seq 77309411337 2026-02-25T22:00:46.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 77309411337' 2026-02-25T22:00:46.886 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:00:47.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411337 -lt 77309411337 2026-02-25T22:00:47.112 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:00:47.112 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:47.113 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:47.397 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:00:47.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:00:47.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-25T22:00:47.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:00:47.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:47.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:47.898 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-25T22:00:47.898 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:00:47.898 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:00:47.898 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-25T22:00:48.094 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:48.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-25T22:00:48.312 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:48.321 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-25T22:00:48.483 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:48.484 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:00:48.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-25T22:00:48.633 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:48.634 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:00:48.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-25T22:00:48.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2026-02-25T22:00:48.851 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:48.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2026-02-25T22:00:48.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2026-02-25T22:00:49.057 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:49.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2026-02-25T22:00:49.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2026-02-25T22:00:49.265 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:49.266 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2026-02-25T22:00:49.266 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2026-02-25T22:00:49.482 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:49.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2026-02-25T22:00:49.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:49.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:49.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:49.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:49.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:49.791 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:49.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2026-02-25T22:00:50.004 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 is already down. 2026-02-25T22:00:50.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2026-02-25T22:00:50.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1731: wait_for_peered: local cmd= 2026-02-25T22:00:50.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1732: wait_for_peered: local num_peered=-1 2026-02-25T22:00:50.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1733: wait_for_peered: local cur_peered 2026-02-25T22:00:50.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: get_timeout_delays 90 .1 2026-02-25T22:00:50.016 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:00:50.016 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:00:50.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:00:50.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:00:50.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:00:50.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:00:50.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: local -a delays 2026-02-25T22:00:50.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1735: wait_for_peered: local -i loop=0 2026-02-25T22:00:50.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: wait_for_peered: flush_pg_stats 2026-02-25T22:00:50.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:00:50.111 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:00:50.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:00:50.318 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T22:00:50.319 INFO:tasks.workunit.client.0.trial163.stderr:2 2026-02-25T22:00:50.319 INFO:tasks.workunit.client.0.trial163.stderr:3' 2026-02-25T22:00:50.319 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:00:50.319 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:50.319 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:00:50.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T22:00:50.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T22:00:50.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T22:00:50.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T22:00:50.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:50.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:00:50.454 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-25T22:00:50.454 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-25T22:00:50.454 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 1-42949672973' 2026-02-25T22:00:50.454 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:50.455 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:00:50.533 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542156 2026-02-25T22:00:50.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542156 2026-02-25T22:00:50.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 1-42949672973 2-60129542156' 2026-02-25T22:00:50.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:00:50.534 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:00:50.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411340 2026-02-25T22:00:50.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411340 2026-02-25T22:00:50.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 1-42949672973 2-60129542156 3-77309411340' 2026-02-25T22:00:50.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:50.612 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672973 2026-02-25T22:00:50.612 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:50.613 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:00:50.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-25T22:00:50.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:50.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672973 2026-02-25T22:00:50.615 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 42949672973 2026-02-25T22:00:50.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-25T22:00:50.615 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:00:50.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672971 -lt 42949672973 2026-02-25T22:00:50.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:51.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:00:51.823 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:00:52.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672971 -lt 42949672973 2026-02-25T22:00:52.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:00:53.040 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:00:53.041 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:00:53.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672974 -lt 42949672973 2026-02-25T22:00:53.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:53.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542156 2026-02-25T22:00:53.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:53.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:00:53.255 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542156 2026-02-25T22:00:53.255 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:53.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542156 2026-02-25T22:00:53.256 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.2 seq 60129542156 2026-02-25T22:00:53.257 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2026-02-25T22:00:53.257 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:00:53.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542157 -lt 60129542156 2026-02-25T22:00:53.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:00:53.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-77309411340 2026-02-25T22:00:53.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:00:53.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:00:53.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411340 2026-02-25T22:00:53.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:00:53.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=77309411340 2026-02-25T22:00:53.473 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.3 seq 77309411340 2026-02-25T22:00:53.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 77309411340' 2026-02-25T22:00:53.473 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:00:53.683 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 77309411340 -lt 77309411340 2026-02-25T22:00:53.683 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: get_num_pgs 2026-02-25T22:00:53.684 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:53.684 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:53.967 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: test 32 == 0 2026-02-25T22:00:53.967 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: wait_for_peered: true 2026-02-25T22:00:53.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: get_num_active_or_peered 2026-02-25T22:00:53.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-25T22:00:53.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-25T22:00:53.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-25T22:00:53.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-25T22:00:53.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-25T22:00:54.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: cur_peered=32 2026-02-25T22:00:54.183 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: get_num_pgs 2026-02-25T22:00:54.183 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:00:54.183 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:00:54.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: test 32 = 32 2026-02-25T22:00:54.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: break 2026-02-25T22:00:54.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_peered: return 0 2026-02-25T22:00:54.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-25T22:00:54.678 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2026-02-25T22:00:54.689 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-25T22:00:54.893 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:54.904 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-25T22:00:55.053 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:55.053 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:00:55.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-25T22:00:55.206 INFO:tasks.workunit.client.0.trial163.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:00:55.206 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:00:55.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:55.212 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:55.212 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:55.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:55.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:55.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:55.322 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:55.322 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:00:55.322 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:55.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:55.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:55.323 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:55.323 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:55.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:55.324 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:55.324 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:55.329 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:55.330 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:55.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:55.331 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:55.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:55.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:55.333 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:55.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:55.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:55.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:55.335 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:55.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:55.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:55.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:55.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:55.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:55.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:55.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:55.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:55.361 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:55.361 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:55.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:55.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:55.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:55.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:55.362 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:55.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:55.363 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:55.364 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:55.364 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:55.365 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:55.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:55.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:55.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:55.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:55.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:55.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:55.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:55.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:55.370 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:55.370 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:55.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:55.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:55.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:55.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:00:55.372 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:00:55.373 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:55.373 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:55.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:00:55.374 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:00:55.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:00:55.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:55.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:00:55.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:00:55.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:55.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:55.397 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:55.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:55.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:55.398 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:55.398 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:55.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:00:55.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:55.426 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:00:55.426 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:55.426 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:55.426 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:55.427 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:55.428 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:55.429 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:55.429 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:00:55.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:00:55.485 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:55.485 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:55.485 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:55.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:55.486 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:55.486 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:55.486 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:55.486 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:55.486 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:55.487 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:55.487 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:55.487 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:55.487 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:00:55.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2026-02-25T22:00:55.649 INFO:tasks.workunit.client.0.trial163.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:00:55.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2026-02-25T22:00:55.759 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2026-02-25T22:00:55.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2026-02-25T22:00:55.847 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-25T22:00:55.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2026-02-25T22:00:55.936 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-25T22:00:55.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2026-02-25T22:00:56.043 INFO:tasks.workunit.client.0.trial163.stderr:mon.dne does not exist 2026-02-25T22:00:56.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:56.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:56.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:56.054 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:56.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:56.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:56.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:56.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:56.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:56.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:56.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:56.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:56.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:56.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:56.170 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:56.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:56.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:56.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:56.172 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:56.173 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:56.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:56.175 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:56.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:56.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.176 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:56.179 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:56.180 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:56.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:56.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:56.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:56.181 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:56.181 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:56.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:56.183 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:56.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:56.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:56.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:56.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:56.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:56.186 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:56.186 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:56.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:56.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:56.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:56.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:56.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:56.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:56.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:56.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:00:56.192 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:00:56.192 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.192 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.192 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:00:56.194 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:00:56.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:00:56.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:00:56.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:00:56.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:56.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:56.217 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:56.218 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:56.218 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.218 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.218 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:56.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:00:56.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:56.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:00:56.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:56.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:56.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:56.248 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:56.248 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:56.249 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:00:56.305 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:00:56.305 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:56.305 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:56.305 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:56.306 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:56.306 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:56.306 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:56.306 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:56.307 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:56.307 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.307 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.307 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:56.307 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:56.307 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:00:56.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:00:56.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:56.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:56.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:56.381 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:56.381 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.381 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:56.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:00:56.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:56.410 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:00:56.410 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:56.410 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:00:56.410 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:56.411 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:56.411 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:00:56.412 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:56.413 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get fsid 2026-02-25T22:00:56.467 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:00:56.468 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:56.468 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:00:56.468 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:56.469 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:56.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:00:56.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:56.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get mon_host 2026-02-25T22:00:56.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:00:56.624 INFO:tasks.workunit.client.0.trial163.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:00:56.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:00:56.734 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2026-02-25T22:00:56.736 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:00:56.822 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2026-02-25T22:00:56.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:00:56.922 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2026-02-25T22:00:56.925 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:00:57.028 INFO:tasks.workunit.client.0.trial163.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:00:57.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:00:57.142 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:00:57.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:00:57.263 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:00:57.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2026-02-25T22:00:57.382 INFO:tasks.workunit.client.0.trial163.stderr:mon.dne does not exist 2026-02-25T22:00:57.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:57.391 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:57.391 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:57.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:57.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:57.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:57.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:57.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:00:57.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:57.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:57.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:57.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:57.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:57.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:57.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:57.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:57.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:57.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:57.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:57.507 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:57.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:57.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:57.509 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:57.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:57.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:57.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:57.510 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:57.511 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:57.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:57.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:57.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:57.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:57.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:57.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:57.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:00:57.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:00:57.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:00:57.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:00:57.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:00:57.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:00:57.520 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:00:57.521 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:00:57.521 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:00:57.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:00:57.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:00:57.523 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:00:57.524 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:57.524 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:00:57.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:00:57.525 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:00:57.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:00:57.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:00:57.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:00:57.528 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:00:57.528 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.528 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:00:57.530 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:00:57.530 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:00:57.530 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:00:57.531 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:00:57.531 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.531 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.531 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:00:57.533 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:00:57.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:00:57.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:00:57.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:00:57.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:00:57.557 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:57.557 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:57.557 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:57.558 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:57.558 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.558 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.558 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:57.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:00:57.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:57.589 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:00:57.589 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:57.589 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:57.589 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:57.590 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:57.590 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:57.591 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:57.592 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:00:57.645 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:00:57.646 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:57.646 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:00:57.646 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.647 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:00:57.648 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:57.648 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:00:57.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:00:57.721 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:57.721 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:57.721 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:57.722 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:57.722 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.722 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.722 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:57.722 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:00:57.748 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:57.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:00:57.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:57.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:00:57.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:57.750 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:57.750 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:00:57.750 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:57.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get fsid 2026-02-25T22:00:57.806 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:00:57.806 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:57.806 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:00:57.806 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:57.806 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:57.807 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:00:57.807 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:57.808 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get mon_host 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-25T22:00:57.861 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-25T22:00:57.879 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:00:57.879 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:00:57.879 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:00:57.880 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:00:57.880 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.880 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.880 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:00:57.880 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-25T22:00:57.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:00:57.910 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-25T22:00:57.910 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:57.910 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:00:57.910 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:00:57.911 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:00:57.911 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:00:57.911 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:00:57.911 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:00:57.911 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:57.912 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.912 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.912 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.c.asok 2026-02-25T22:00:57.912 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:57.912 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.c.asok config get fsid 2026-02-25T22:00:57.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-25T22:00:57.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:00:57.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:00:57.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.c.asok 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:00:57.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.c.asok config get mon_host 2026-02-25T22:00:58.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2026-02-25T22:00:58.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:00:58.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-25T22:00:58.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:00:58.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-25T22:00:58.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:00:58.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:00:58.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772056918 2026-02-25T22:00:58.026 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:00:58.027 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056858 -lt 1772056918 ]] 2026-02-25T22:00:58.027 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:00:58.028 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:04.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:04.142 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:04.143 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:04.143 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:04.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:01:04.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:01:04.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:01:04.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:01:04.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:01:04.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:01:04.254 INFO:tasks.workunit.client.0.trial163.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:01:04.262 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:01:04.367 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c) after stopping [a] 2026-02-25T22:01:04.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:01:07.487 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-25T22:01:07.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:01:07.612 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b) after stopping [c] 2026-02-25T22:01:07.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:01:07.705 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2026-02-25T22:01:07.709 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2026-02-25T22:01:07.797 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2026-02-25T22:01:07.801 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2026-02-25T22:01:07.888 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2026-02-25T22:01:07.892 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:01:07.996 INFO:tasks.workunit.client.0.trial163.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:01:08.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:01:11.146 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:01:11.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:01:11.268 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:01:11.276 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:01:11.380 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.c 2026-02-25T22:01:11.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2026-02-25T22:01:11.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:01:11.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:01:11.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:01:11.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:01:11.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:01:11.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:01:11.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772056931 2026-02-25T22:01:11.499 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:11.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056871 -lt 1772056931 ]] 2026-02-25T22:01:11.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:11.501 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:11.607 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:11.607 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.608 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:11.608 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:11.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:11.618 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:11.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056871 -lt 1772056931 ]] 2026-02-25T22:01:11.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:11.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:11.731 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:11.732 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.732 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:11.733 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:11.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:11.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:11.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056871 -lt 1772056931 ]] 2026-02-25T22:01:11.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:11.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:11.853 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:11.853 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.854 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:11.854 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:11.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:11.864 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:11.865 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056871 -lt 1772056931 ]] 2026-02-25T22:01:11.865 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:11.866 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:11.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:11.979 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.980 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:11.980 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:11.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:11.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:11.989 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:11.990 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056871 -lt 1772056931 ]] 2026-02-25T22:01:11.990 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:11.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:12.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:12.105 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:12.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:12.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:12.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:12.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:12.116 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:12.117 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056872 -lt 1772056931 ]] 2026-02-25T22:01:12.118 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:12.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:15.236 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:15.236 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:15.237 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:15.237 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:15.245 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:15.245 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:15.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:15.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056875 -lt 1772056931 ]] 2026-02-25T22:01:15.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:15.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:15.364 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:15.365 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:15.365 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:15.365 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:15.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:15.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:15.376 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:15.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056875 -lt 1772056931 ]] 2026-02-25T22:01:15.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:15.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:18.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:18.488 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.488 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:18.489 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.497 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:18.497 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:18.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:18.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056878 -lt 1772056931 ]] 2026-02-25T22:01:18.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:18.499 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:18.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:18.609 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.610 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:18.610 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.620 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:18.620 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:18.620 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:18.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056878 -lt 1772056931 ]] 2026-02-25T22:01:18.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:18.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:18.735 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:18.735 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.735 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.736 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:18.745 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:18.745 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:18.745 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:18.746 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056878 -lt 1772056931 ]] 2026-02-25T22:01:18.746 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:18.747 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:18.860 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:18.860 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.861 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.861 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:18.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:18.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:18.870 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:18.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056878 -lt 1772056931 ]] 2026-02-25T22:01:18.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:18.872 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:18.980 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:18.980 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.981 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:18.981 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:18.992 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:18.992 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:18.992 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:18.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056878 -lt 1772056931 ]] 2026-02-25T22:01:18.993 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:18.994 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:19.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:19.108 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:19.109 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:19.109 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:19.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:19.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:19.120 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:19.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056879 -lt 1772056931 ]] 2026-02-25T22:01:19.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:19.121 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:19.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:19.233 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:19.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:19.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:19.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:19.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:19.245 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:19.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056879 -lt 1772056931 ]] 2026-02-25T22:01:19.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:01:19.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:26.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:26.763 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:26.764 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:01:26.764 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:00:57.549973Z","created":"2026-02-25T22:00:57.549973Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549969+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549970+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:00:57:549971+0000","crush_location":[]}]}}' 2026-02-25T22:01:26.773 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:01:26.773 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:01:26.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:01:26.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:01:26.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:01:26.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:01:26.860 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2026-02-25T22:01:26.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:01:29.971 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-25T22:01:29.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:01:30.076 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2026-02-25T22:01:30.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:01:30.167 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-25T22:01:30.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:01:30.254 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-25T22:01:30.258 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:01:30.363 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:01:30.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:01:30.472 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-25T22:01:30.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:01:30.477 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:01:30.477 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:01:30.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:01:30.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:01:30.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:01:30.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:01:30.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:01:30.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:01:30.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:01:30.592 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:01:30.593 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:01:30.593 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:01:30.594 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:01:30.595 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:01:30.595 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:01:30.596 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:01:30.596 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:01:30.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:01:30.597 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:01:30.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:01:30.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:01:30.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:01:30.606 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:01:30.606 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.606 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:01:30.609 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:01:30.610 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:01:30.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:01:30.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:01:30.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:01:30.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:01:30.611 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:01:30.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:01:30.612 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:01:30.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:01:30.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:01:30.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:01:30.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:01:30.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:01:30.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:01:30.616 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:01:30.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:01:30.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:01:30.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:01:30.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:01:30.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.620 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:01:30.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:01:30.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:01:30.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:01:30.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:01:30.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.622 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:01:30.624 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:01:30.625 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:01:30.625 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:01:30.625 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:01:30.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:01:30.647 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:01:30.647 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:01:30.647 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:01:30.648 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:01:30.648 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.648 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.648 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:01:30.648 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:01:30.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:01:30.676 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:01:30.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:30.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:01:30.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.678 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:01:30.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:30.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:01:30.735 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:01:30.735 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:30.735 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:01:30.736 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.737 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:01:30.738 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:30.738 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:01:30.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:01:30.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:01:30.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:01:30.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:01:30.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:01:30.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:01:30.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:01:30.812 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:01:30.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:01:30.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:01:30.814 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:01:30.814 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.814 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:01:30.814 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:01:30.841 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:01:30.842 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:01:30.842 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:30.842 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:01:30.843 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:01:30.843 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:01:30.844 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:30.845 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get fsid 2026-02-25T22:01:30.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:01:30.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:30.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:01:30.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:01:30.901 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.902 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:01:30.903 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:30.903 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get mon_host 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-25T22:01:30.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-25T22:01:30.978 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:01:30.978 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:01:30.978 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:01:30.979 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:01:30.979 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:30.979 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:30.979 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:01:30.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-25T22:01:31.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:01:31.005 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-25T22:01:31.006 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:31.006 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:01:31.006 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:31.007 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.c.asok 2026-02-25T22:01:31.008 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:31.008 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.c.asok config get fsid 2026-02-25T22:01:31.064 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-25T22:01:31.064 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:31.064 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:01:31.064 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:01:31.065 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:01:31.065 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:01:31.065 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.c.asok 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:31.066 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.c.asok config get mon_host 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-25T22:01:31.121 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-25T22:01:31.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:01:31.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:01:31.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:01:31.143 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:01:31.143 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:31.143 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:31.144 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:01:31.144 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-25T22:01:31.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:01:31.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-25T22:01:31.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:31.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:01:31.173 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:01:31.174 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:31.175 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.d.asok 2026-02-25T22:01:31.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:31.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.d.asok config get fsid 2026-02-25T22:01:31.232 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-25T22:01:31.232 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:01:31.232 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:01:31.232 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:01:31.233 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:01:31.233 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.d.asok 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:01:31.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.d.asok config get mon_host 2026-02-25T22:01:31.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:01:31.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772056951 2026-02-25T22:01:31.292 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:31.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056891 -lt 1772056951 ]] 2026-02-25T22:01:31.293 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:01:31.293 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:34.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:34.414 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:34.415 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:01:34.415 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:01:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:01:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:01:34.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:01:34.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:01:34.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:01:34.529 INFO:tasks.workunit.client.0.trial163.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:01:34.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:01:34.638 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-25T22:01:34.645 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:01:34.751 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,c,d) after stopping [b] 2026-02-25T22:01:34.759 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:01:34.879 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b,d) after stopping [c] 2026-02-25T22:01:34.887 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:01:35.008 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b,c) after stopping [d] 2026-02-25T22:01:35.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:01:35.112 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2026-02-25T22:01:35.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2026-02-25T22:01:35.203 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2026-02-25T22:01:35.207 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:01:35.310 INFO:tasks.workunit.client.0.trial163.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:01:35.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:01:35.425 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:01:35.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:01:35.540 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:01:35.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:01:35.657 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.c 2026-02-25T22:01:35.665 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-25T22:01:35.666 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:01:35.666 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:01:35.666 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:01:35.666 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:01:35.666 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:01:35.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:01:35.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2026-02-25T22:01:35.774 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:01:35.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-25T22:01:35.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:01:35.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-25T22:01:35.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:01:35.775 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:01:35.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772056955 2026-02-25T22:01:35.776 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:35.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056895 -lt 1772056955 ]] 2026-02-25T22:01:35.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:35.777 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:35.892 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:35.892 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:35.893 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:35.893 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:35.903 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:35.903 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:35.903 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:35.905 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056895 -lt 1772056955 ]] 2026-02-25T22:01:35.905 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:35.905 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:39.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:39.019 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.019 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.019 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:39.028 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:39.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:39.029 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:39.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056899 -lt 1772056955 ]] 2026-02-25T22:01:39.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:39.031 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:39.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:39.148 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:39.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:39.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:39.160 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:39.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056899 -lt 1772056955 ]] 2026-02-25T22:01:39.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:39.161 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:39.275 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:39.275 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.276 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:39.276 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.285 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:39.285 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:39.285 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:39.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056899 -lt 1772056955 ]] 2026-02-25T22:01:39.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:39.287 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:39.398 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:39.398 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.399 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:39.399 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.408 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:39.408 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:39.409 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:39.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056899 -lt 1772056955 ]] 2026-02-25T22:01:39.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:39.410 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:39.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:39.524 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.524 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:39.525 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:39.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:39.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:39.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:39.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056899 -lt 1772056955 ]] 2026-02-25T22:01:39.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:39.537 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:42.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:42.652 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:42.653 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:42.653 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:42.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:42.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:42.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:42.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056902 -lt 1772056955 ]] 2026-02-25T22:01:42.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:42.664 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:42.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:42.779 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:42.780 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:42.780 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:42.790 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:42.791 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:42.791 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:42.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056902 -lt 1772056955 ]] 2026-02-25T22:01:42.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:42.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:42.903 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:42.903 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:42.904 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:42.904 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:42.915 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:01:42.915 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:01:42.915 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:01:42.917 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056902 -lt 1772056955 ]] 2026-02-25T22:01:42.917 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:01:42.917 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:01:56.449 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:01:56.450 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:56.450 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:01:56.450 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:01:30.641104Z","created":"2026-02-25T22:01:30.641104Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641100+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641102+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:01:30:641103+0000","crush_location":[]}]}}' 2026-02-25T22:01:56.460 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:01:56.460 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:01:56.460 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:01:56.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:01:56.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:01:56.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:01:56.564 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-25T22:01:56.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:01:59.664 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-25T22:01:59.666 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:01:59.757 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-25T22:01:59.759 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:01:59.845 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-25T22:01:59.849 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:01:59.955 INFO:tasks.workunit.client.0.trial163.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:01:59.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:02:00.083 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:02:00.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:02:00.208 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:02:00.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:02:00.332 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.c 2026-02-25T22:02:00.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:02:00.458 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.d 2026-02-25T22:02:00.466 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:02:00.467 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:02:00.467 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:02:00.467 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:02:00.467 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:02:00.467 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:02:00.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:02:00.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:02:00.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:02:00.582 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:02:00.582 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:02:00.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:02:00.583 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:02:00.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:02:00.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:02:00.585 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:02:00.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:02:00.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:02:00.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:02:00.587 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:02:00.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:02:00.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:02:00.589 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:02:00.597 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:02:00.597 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.598 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:02:00.600 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:02:00.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:02:00.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:02:00.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:02:00.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:02:00.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:02:00.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:02:00.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:02:00.605 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:02:00.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:02:00.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:02:00.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:02:00.607 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:02:00.608 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:02:00.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:02:00.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:02:00.610 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:02:00.610 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.610 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:02:00.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:02:00.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:02:00.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:02:00.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:02:00.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.613 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.36032 2026-02-25T22:02:00.615 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:02:00.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:02:00.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=2ff0436b-bdee-4e25-8a10-656edb8b6af6 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:02:00.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:02:00.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:02:00.667 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:02:00.668 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:02:00.668 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:00.668 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:02:00.668 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:02:00.669 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:02:00.669 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:00.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get fsid 2026-02-25T22:02:00.725 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:02:00.725 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:00.725 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:02:00.725 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:02:00.725 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:02:00.726 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.a.asok 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:00.727 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.a.asok config get mon_host 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:02:00.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:02:00.801 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:02:00.801 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:02:00.801 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:02:00.801 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:02:00.801 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.801 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:02:00.802 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:02:00.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:02:00.827 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:02:00.827 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:00.827 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:02:00.827 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.829 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:02:00.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:00.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get fsid 2026-02-25T22:02:00.884 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:02:00.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:00.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:02:00.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:02:00.886 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:02:00.886 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:02:00.886 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:02:00.886 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:02:00.886 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:00.887 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.887 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.887 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.b.asok 2026-02-25T22:02:00.887 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:00.887 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.b.asok config get mon_host 2026-02-25T22:02:00.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-25T22:02:00.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:02:00.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:02:00.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-25T22:02:00.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:02:00.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-25T22:02:00.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-25T22:02:00.961 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:02:00.961 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:02:00.961 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:02:00.961 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:02:00.962 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.962 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.962 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:02:00.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-25T22:02:00.990 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:02:00.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-25T22:02:00.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:00.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:02:00.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:02:00.992 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:02:00.992 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:02:00.992 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:02:00.992 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:02:00.992 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:00.992 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:00.993 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:00.993 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.c.asok 2026-02-25T22:02:00.993 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:00.993 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.c.asok config get fsid 2026-02-25T22:02:01.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-25T22:02:01.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:01.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:02:01.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:02:01.049 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:02:01.049 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:02:01.050 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:02:01.050 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:02:01.051 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:01.051 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.051 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.051 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.c.asok 2026-02-25T22:02:01.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:01.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.c.asok config get mon_host 2026-02-25T22:02:01.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-25T22:02:01.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:02:01.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:02:01.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-25T22:02:01.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:02:01.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-25T22:02:01.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-25T22:02:01.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:02:01.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:02:01.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:02:01.125 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:02:01.125 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.125 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.126 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:02:01.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-25T22:02:01.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:02:01.154 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-25T22:02:01.154 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:01.154 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:02:01.154 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:02:01.154 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:02:01.155 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:02:01.155 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.d.asok 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:01.156 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.d.asok config get fsid 2026-02-25T22:02:01.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-25T22:02:01.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:01.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:02:01.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:02:01.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.d.asok 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:01.213 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.d.asok config get mon_host 2026-02-25T22:02:01.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2026-02-25T22:02:01.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:02:01.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:02:01.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2026-02-25T22:02:01.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:02:01.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/e 2026-02-25T22:02:01.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2026-02-25T22:02:01.291 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:02:01.291 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:02:01.291 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:02:01.291 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:02:01.291 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.292 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.292 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.36032/$cluster-$name.asok' 2026-02-25T22:02:01.292 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.36032/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2026-02-25T22:02:01.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:02:01.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2026-02-25T22:02:01.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:01.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-25T22:02:01.329 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:02:01.329 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.e.asok 2026-02-25T22:02:01.330 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:01.331 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.e.asok config get fsid 2026-02-25T22:02:01.384 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2026-02-25T22:02:01.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:02:01.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-25T22:02:01.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:02:01.385 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:02:01.386 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-25T22:02:01.386 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-25T22:02:01.386 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-25T22:02:01.386 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:02:01.386 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:02:01.386 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:02:01.387 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.36032/ceph-mon.e.asok 2026-02-25T22:02:01.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:02:01.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.36032/ceph-mon.e.asok config get mon_host 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:02:01.441 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:02:01.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772056981 2026-02-25T22:02:01.442 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:01.443 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056921 -lt 1772056981 ]] 2026-02-25T22:02:01.443 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-25T22:02:01.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:02.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:02.966 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:02.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2026-02-25T22:02:02.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:02.976 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:02:02.977 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:02:02.977 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:02:02.977 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:02:02.977 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:02:02.977 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:02:03.078 INFO:tasks.workunit.client.0.trial163.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:02:03.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:02:03.204 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-25T22:02:03.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:02:03.317 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2026-02-25T22:02:03.325 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:02:06.446 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2026-02-25T22:02:06.456 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:02:06.558 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2026-02-25T22:02:06.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-25T22:02:06.668 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2026-02-25T22:02:06.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:02:06.783 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2026-02-25T22:02:06.791 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2026-02-25T22:02:06.907 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2026-02-25T22:02:06.917 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2026-02-25T22:02:07.002 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2026-02-25T22:02:07.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:02:07.108 INFO:tasks.workunit.client.0.trial163.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:02:07.118 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:02:07.220 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:02:07.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:02:10.361 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:02:10.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:02:10.478 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.c 2026-02-25T22:02:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:02:13.589 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.d 2026-02-25T22:02:13.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-25T22:02:13.707 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.e 2026-02-25T22:02:13.714 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-25T22:02:13.715 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:02:13.715 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:02:13.716 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:02:13.716 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:02:13.716 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:02:13.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:02:13.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:02:13.825 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772056993 2026-02-25T22:02:13.825 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:13.827 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056933 -lt 1772056993 ]] 2026-02-25T22:02:13.827 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:13.827 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:13.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:13.941 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:13.942 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:13.942 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:13.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:13.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:13.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:13.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056933 -lt 1772056993 ]] 2026-02-25T22:02:13.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:13.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.073 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.084 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.086 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.201 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.201 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.202 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.202 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.213 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.214 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.329 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.329 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.330 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.330 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.341 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.341 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.341 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.342 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.342 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.455 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.456 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.456 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.467 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.467 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.467 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.468 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.579 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.579 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.580 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.580 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.590 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.590 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.590 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.592 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.705 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.705 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.716 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.716 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.717 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.718 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.718 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.718 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.834 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.834 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.835 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.835 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.845 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.845 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.845 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.847 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.847 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.847 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:17.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:17.963 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.963 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:17.963 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:17.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:17.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:17.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:17.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056937 -lt 1772056993 ]] 2026-02-25T22:02:17.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:17.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:18.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:18.088 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:18.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:18.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:18.098 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:18.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056938 -lt 1772056993 ]] 2026-02-25T22:02:18.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:18.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:18.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:18.215 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.216 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.216 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:18.225 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:18.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:18.226 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:18.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056938 -lt 1772056993 ]] 2026-02-25T22:02:18.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:18.227 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:18.342 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:18.342 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:18.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:18.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:18.354 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:18.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056938 -lt 1772056993 ]] 2026-02-25T22:02:18.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:18.355 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:18.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:18.470 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:18.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:18.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:18.481 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:18.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056938 -lt 1772056993 ]] 2026-02-25T22:02:18.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:18.482 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:18.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:18.591 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.592 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:18.592 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:18.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:18.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:18.602 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:18.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056938 -lt 1772056993 ]] 2026-02-25T22:02:18.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:02:18.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:29.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:29.135 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:29.136 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:29.136 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:02:29.145 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:02:29.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:02:29.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:02:29.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:02:29.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:02:29.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:02:29.252 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-25T22:02:29.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:02:29.383 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (c,d,e) after stopping [b] 2026-02-25T22:02:29.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:02:32.510 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,d,e) after stopping [c] 2026-02-25T22:02:32.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:02:32.631 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,e) after stopping [d] 2026-02-25T22:02:32.642 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-25T22:02:32.750 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-25T22:02:32.758 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:02:32.873 INFO:tasks.workunit.client.0.trial163.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:02:32.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:02:32.999 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:02:33.007 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:02:33.125 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.b 2026-02-25T22:02:33.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:02:33.254 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.c 2026-02-25T22:02:33.262 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:02:33.376 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.d 2026-02-25T22:02:33.387 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-25T22:02:33.493 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.e 2026-02-25T22:02:33.501 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2026-02-25T22:02:33.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:02:33.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:02:33.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:02:33.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:02:33.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:02:33.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:02:33.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2026-02-25T22:02:33.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:02:33.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-25T22:02:33.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:02:33.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-25T22:02:33.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:02:33.611 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:02:33.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772057013 2026-02-25T22:02:33.611 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:33.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056953 -lt 1772057013 ]] 2026-02-25T22:02:33.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:33.612 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:33.727 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:33.727 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:33.728 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:33.728 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:33.737 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:33.737 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:33.737 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:33.738 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056953 -lt 1772057013 ]] 2026-02-25T22:02:33.738 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:33.738 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:33.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:33.852 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:33.852 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:33.853 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:33.862 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:33.862 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:33.862 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:33.863 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056953 -lt 1772057013 ]] 2026-02-25T22:02:33.863 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:33.864 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:36.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:36.978 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:36.979 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:36.979 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:36.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:36.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:36.989 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:36.990 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056956 -lt 1772057013 ]] 2026-02-25T22:02:36.990 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:36.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:37.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:37.106 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:37.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:37.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:37.117 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:37.117 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:37.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:37.118 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056957 -lt 1772057013 ]] 2026-02-25T22:02:37.118 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:37.119 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:37.232 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:37.232 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:37.233 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:37.233 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:37.242 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:37.242 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:37.243 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:37.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056957 -lt 1772057013 ]] 2026-02-25T22:02:37.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:37.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:40.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:40.355 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:40.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:40.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:40.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:40.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056960 -lt 1772057013 ]] 2026-02-25T22:02:40.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:40.368 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:40.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:40.481 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.481 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:40.482 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.490 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:40.490 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:40.491 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:40.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056960 -lt 1772057013 ]] 2026-02-25T22:02:40.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:40.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:40.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:40.604 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.604 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.604 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:40.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:40.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:40.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:40.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056960 -lt 1772057013 ]] 2026-02-25T22:02:40.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:40.616 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:40.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:40.730 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.730 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:40.730 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:40.740 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:02:40.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:02:40.741 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:02:40.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772056960 -lt 1772057013 ]] 2026-02-25T22:02:40.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:02:40.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:02:52.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:02:52.858 INFO:tasks.workunit.client.0.trial163.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:52.859 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"2ff0436b-bdee-4e25-8a10-656edb8b6af6","modified":"2026-02-25T22:02:00.631213Z","created":"2026-02-25T22:02:00.631213Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631208+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631210+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631211+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:02:00:631212+0000","crush_location":[]}]}}' 2026-02-25T22:02:52.859 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:02:52.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:02:52.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:02:52.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:02:52.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:02:52.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:02:52.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:02:52.970 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-25T22:02:52.977 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:02:53.076 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-25T22:02:53.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:02:53.163 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-25T22:02:53.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:02:56.252 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-25T22:02:56.255 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-25T22:02:59.358 INFO:tasks.workunit.client.0.trial163.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-25T22:02:59.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:03:02.459 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-25T22:03:02.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:03:02.564 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.a 2026-02-25T22:03:02.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:03:02.670 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: removing mon.b would break quorum 2026-02-25T22:03:02.674 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:03:05.760 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-25T22:03:05.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:03:05.847 INFO:tasks.workunit.client.0.trial163.stderr:Error EBUSY: removing mon.d would break quorum 2026-02-25T22:03:05.850 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-25T22:03:08.957 INFO:tasks.workunit.client.0.trial163.stderr:safe to remove mon.e 2026-02-25T22:03:08.967 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:03:08.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:08.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:08.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:08.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:08.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:09.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:09.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:03:09.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:03:09.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:09.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:03:09.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:09.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:09.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:09.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:09.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:09.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:09.083 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:09.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:09.084 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:09.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:09.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:09.085 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:09.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:09.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:09.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:09.087 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:09.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:09.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:09.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:03:09.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:09.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:03:09.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/ok-to-stop 0 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:03:09.103 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:09.104 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:09.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:09.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:09.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:09.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:09.106 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:09.106 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:09.107 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:09.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:09.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:09.109 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:09.110 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:09.110 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:09.110 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:09.111 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:09.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:09.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:03:09.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:03:09.114 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:09.114 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.114 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.36032 2026-02-25T22:03:09.115 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.36032 2026-02-25T22:03:09.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:09.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:09.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:03:09.117 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:03:09.117 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:03:09.171 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2026-02-25T22:03:09.171 DEBUG:teuthology.orchestra.run.trial163:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2026-02-25T22:03:09.224 INFO:tasks.workunit.client.0.trial163.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:03:09.228 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/rados-striper 2026-02-25T22:03:09.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2026-02-25T22:03:09.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2026-02-25T22:03:09.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2026-02-25T22:03:09.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2026-02-25T22:03:09.229 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2026-02-25T22:03:09.229 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2026-02-25T22:03:09.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none ' 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/rados-striper 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/rados-striper 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:09.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:09.232 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:09.232 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:09.232 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:09.232 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:09.233 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:09.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:09.234 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:09.234 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:09.235 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:09.235 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:09.235 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:09.236 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:09.236 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:09.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:09.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-25T22:03:09.239 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:09.239 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.240 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:09.240 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.51242 2026-02-25T22:03:09.241 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:09.241 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:09.241 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/rados-striper 2026-02-25T22:03:09.242 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:09.243 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.243 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:09.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.51242 2026-02-25T22:03:09.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:09.245 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:03:09.245 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/rados-striper 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/rados-striper/a 2026-02-25T22:03:09.246 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2026-02-25T22:03:09.268 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:09.268 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:09.268 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:09.269 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:09.269 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.269 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:09.269 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:09.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:09.298 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:09.299 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:09.299 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:09.299 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:09.299 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:09.299 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:09.300 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:09.300 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:09.300 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:09.300 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:09.301 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.301 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:09.301 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.51242/ceph-mon.a.asok 2026-02-25T22:03:09.301 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:09.301 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.51242/ceph-mon.a.asok config get fsid 2026-02-25T22:03:09.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:09.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:09.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:09.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:09.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:09.357 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:09.357 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:09.357 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:09.358 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:09.358 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.358 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:09.358 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.51242/ceph-mon.a.asok 2026-02-25T22:03:09.358 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:09.358 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.51242/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/0 2026-02-25T22:03:09.417 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:09.418 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:09.419 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:09.420 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:09.420 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:09.420 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/0 2026-02-25T22:03:09.420 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:09.421 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=214e0f8e-74cd-403e-8cce-f8ab2d63dc72 2026-02-25T22:03:09.421 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 214e0f8e-74cd-403e-8cce-f8ab2d63dc72' 2026-02-25T22:03:09.421 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 214e0f8e-74cd-403e-8cce-f8ab2d63dc72 2026-02-25T22:03:09.421 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:09.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdcZ9p2JywGRAAGCBtSX34+uPAddH5ziCXEg== 2026-02-25T22:03:09.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdcZ9p2JywGRAAGCBtSX34+uPAddH5ziCXEg=="}' 2026-02-25T22:03:09.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 214e0f8e-74cd-403e-8cce-f8ab2d63dc72 -i td/rados-striper/0/new.json 2026-02-25T22:03:09.552 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:09.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/0/new.json 2026-02-25T22:03:09.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdcZ9p2JywGRAAGCBtSX34+uPAddH5ziCXEg== --osd-uuid 214e0f8e-74cd-403e-8cce-f8ab2d63dc72 2026-02-25T22:03:09.582 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:09.579+0000 7f78112a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:09.584 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:09.579+0000 7f78112a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:09.585 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:09.579+0000 7f78112a2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:09.586 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:09.579+0000 7f78112a2a40 -1 bdev(0x56037cfe8400 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:09.586 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:09.579+0000 7f78112a2a40 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2026-02-25T22:03:10.252 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/0/keyring 2026-02-25T22:03:10.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:10.254 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:10.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:10.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:10.379 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:10.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:10.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:10.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:10.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:10.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:10.395 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:10.391+0000 7f77ec3cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:10.396 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:10.391+0000 7f77ec3cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:10.397 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:10.391+0000 7f77ec3cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:10.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:10.486 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:10.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:10.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:10.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:10.596 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:10.740 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:10.735+0000 7f77ec3cfa40 -1 Falling back to public interface 2026-02-25T22:03:11.362 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:11.355+0000 7f77ec3cfa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:11.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:11.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:11.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:11.598 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:11.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:11.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:11.707 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:12.708 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:12.708 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:12.709 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:12.709 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:12.709 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:12.710 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:12.816 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3112935710,v1:127.0.0.1:6801/3112935710] [v2:127.0.0.1:6802/3112935710,v1:127.0.0.1:6803/3112935710] exists,up 214e0f8e-74cd-403e-8cce-f8ab2d63dc72 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:03:12.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/1 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:12.821 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:12.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:12.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:12.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:12.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:12.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:12.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/1 2026-02-25T22:03:12.823 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:12.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c3dc79ea-2ed0-4c79-8709-3508a92d663a 2026-02-25T22:03:12.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c3dc79ea-2ed0-4c79-8709-3508a92d663a' 2026-02-25T22:03:12.824 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 c3dc79ea-2ed0-4c79-8709-3508a92d663a 2026-02-25T22:03:12.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:12.832 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgcZ9poAeMMRAAKxQn+VlBbqaKi6BaZcMuzA== 2026-02-25T22:03:12.832 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgcZ9poAeMMRAAKxQn+VlBbqaKi6BaZcMuzA=="}' 2026-02-25T22:03:12.832 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c3dc79ea-2ed0-4c79-8709-3508a92d663a -i td/rados-striper/1/new.json 2026-02-25T22:03:12.950 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:12.961 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/1/new.json 2026-02-25T22:03:12.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgcZ9poAeMMRAAKxQn+VlBbqaKi6BaZcMuzA== --osd-uuid c3dc79ea-2ed0-4c79-8709-3508a92d663a 2026-02-25T22:03:12.984 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:12.979+0000 7fe57ae3ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:12.986 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:12.979+0000 7fe57ae3ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:12.987 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:12.983+0000 7fe57ae3ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:12.987 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:12.983+0000 7fe57ae3ba40 -1 bdev(0x562b2fdfe400 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:12.987 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:12.983+0000 7fe57ae3ba40 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2026-02-25T22:03:13.630 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/1/keyring 2026-02-25T22:03:13.630 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:13.631 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T22:03:13.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:03:13.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:13.750 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T22:03:13.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:03:13.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:13.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:13.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:13.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:13.772 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:13.767+0000 7f18a1984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:13.774 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:13.771+0000 7f18a1984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:13.776 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:13.771+0000 7f18a1984a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:13.856 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:03:13.856 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:13.856 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:03:13.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:13.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:13.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:13.857 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:13.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:13.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:13.858 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:03:13.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:14.136 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:14.131+0000 7f18a1984a40 -1 Falling back to public interface 2026-02-25T22:03:14.721 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:14.715+0000 7f18a1984a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:03:14.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:14.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:14.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:14.971 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:14.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:14.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:03:15.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:16.080 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:16.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:16.081 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:16.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:16.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:16.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:03:16.191 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2464620832,v1:127.0.0.1:6809/2464620832] [v2:127.0.0.1:6810/2464620832,v1:127.0.0.1:6811/2464620832] exists,up c3dc79ea-2ed0-4c79-8709-3508a92d663a 2026-02-25T22:03:16.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:16.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:16.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:16.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2026-02-25T22:03:16.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/2 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2026-02-25T22:03:16.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:16.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:16.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/2 2026-02-25T22:03:16.198 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:16.198 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a88fb60-d619-43d8-8f28-42474c1c93c1 2026-02-25T22:03:16.198 INFO:tasks.workunit.client.0.trial163.stdout:add osd2 7a88fb60-d619-43d8-8f28-42474c1c93c1 2026-02-25T22:03:16.199 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7a88fb60-d619-43d8-8f28-42474c1c93c1' 2026-02-25T22:03:16.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:16.208 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCkcZ9ph81UDBAAPzNmAnqZtS2MN4I+kP0/4w== 2026-02-25T22:03:16.208 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCkcZ9ph81UDBAAPzNmAnqZtS2MN4I+kP0/4w=="}' 2026-02-25T22:03:16.208 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a88fb60-d619-43d8-8f28-42474c1c93c1 -i td/rados-striper/2/new.json 2026-02-25T22:03:16.324 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:16.332 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/2/new.json 2026-02-25T22:03:16.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkcZ9ph81UDBAAPzNmAnqZtS2MN4I+kP0/4w== --osd-uuid 7a88fb60-d619-43d8-8f28-42474c1c93c1 2026-02-25T22:03:16.348 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:16.343+0000 7ffa6eb7ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:16.349 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:16.343+0000 7ffa6eb7ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:16.351 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:16.347+0000 7ffa6eb7ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:16.351 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:16.347+0000 7ffa6eb7ba40 -1 bdev(0x55612cf76400 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:16.351 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:16.347+0000 7ffa6eb7ba40 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2026-02-25T22:03:16.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/2/keyring 2026-02-25T22:03:16.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:16.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:03:16.999 INFO:tasks.workunit.client.0.trial163.stdout:adding osd2 key to auth repository 2026-02-25T22:03:17.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:17.118 INFO:tasks.workunit.client.0.trial163.stdout:start osd.2 2026-02-25T22:03:17.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:03:17.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=81c49b2f-f5f3-4454-88e8-8f72a288f3eb --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.51242/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:17.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:17.120 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:17.120 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:17.140 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:17.135+0000 7f30e107ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:17.141 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:17.135+0000 7f30e107ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:17.142 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:17.139+0000 7f30e107ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:17.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:03:17.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:17.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:03:17.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:17.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:17.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:17.231 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:17.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:17.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:17.231 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:03:17.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:17.480 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:17.475+0000 7f30e107ea40 -1 Falling back to public interface 2026-02-25T22:03:18.028 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:18.023+0000 7f30e107ea40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:03:18.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:18.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:18.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:18.340 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:18.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:18.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:03:18.453 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:19.454 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:19.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:19.455 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:19.456 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:19.456 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:19.456 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:03:19.565 INFO:tasks.workunit.client.0.trial163.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/3577839722,v1:127.0.0.1:6817/3577839722] [v2:127.0.0.1:6818/3577839722,v1:127.0.0.1:6819/3577839722] exists,up 7a88fb60-d619-43d8-8f28-42474c1c93c1 2026-02-25T22:03:19.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:19.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:19.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:19.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2026-02-25T22:03:19.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:03:19.668 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:03:19.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:03:19.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:03:19.811 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:03:19.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:03:20.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:03:21.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2026-02-25T22:03:21.101 INFO:tasks.workunit.client.0.trial163.stderr:1+0 records in 2026-02-25T22:03:21.101 INFO:tasks.workunit.client.0.trial163.stderr:1+0 records out 2026-02-25T22:03:21.101 INFO:tasks.workunit.client.0.trial163.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.5233e-05 s, 22.3 MB/s 2026-02-25T22:03:21.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2026-02-25T22:03:21.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2026-02-25T22:03:21.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2026-02-25T22:03:21.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:03:21.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2026-02-25T22:03:21.169 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2026-02-25T22:03:21.169 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2026-02-25T22:03:21.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2026-02-25T22:03:21.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2026-02-25T22:03:21.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2026-02-25T22:03:21.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2026-02-25T22:03:21.221 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2026-02-25T22:03:21.222 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2026-02-25T22:03:21.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2026-02-25T22:03:21.245 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2026-02-25T22:03:21.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2026-02-25T22:03:21.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2026-02-25T22:03:21.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:03:21.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2026-02-25T22:03:21.307 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2026-02-25T22:03:21.307 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2026-02-25T22:03:21.308 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2026-02-25T22:03:21.309 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2026-02-25T22:03:21.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-25T22:03:21.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-25T22:03:21.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2026-02-25T22:03:21.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2026-02-25T22:03:21.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2026-02-25T22:03:21.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2026-02-25T22:03:21.406 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2026-02-25T22:03:21.406 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2026-02-25T22:03:21.408 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2026-02-25T22:03:21.408 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2026-02-25T22:03:21.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2026-02-25T22:03:21.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2026-02-25T22:03:21.454 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2026-02-25T22:03:21.455 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No data available' 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:21.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-25T22:03:21.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:03:21.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:03:21.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-25T22:03:21.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:03:21.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-25T22:03:21.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:03:21.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:21.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No data available' 2026-02-25T22:03:21.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:21.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:21.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-25T22:03:21.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:03:21.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:03:21.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-25T22:03:21.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:03:21.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2026-02-25T22:03:21.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2026-02-25T22:03:21.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:03:21.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:21.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No such file or directory' 2026-02-25T22:03:21.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:21.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:21.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd --striper stat toyfile 2026-02-25T22:03:21.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:03:21.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:03:21.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-25T22:03:21.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:03:21.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:03:21.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:03:21.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:21.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No such file or directory' 2026-02-25T22:03:21.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:21.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:21.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:03:21.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:03:21.563 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:03:21.563 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-25T22:03:21.563 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:03:21.564 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2026-02-25T22:03:21.564 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-25T22:03:21.564 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:21.564 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-25T22:03:21.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:21.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:21.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:21.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:21.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:21.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:21.673 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:21.674 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:21.675 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:21.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:21.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:21.676 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:21.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:21.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:21.678 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:21.678 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:21.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:21.680 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:21.680 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-25T22:03:21.688 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:21.688 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.688 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:21.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.51242 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/rados-striper 0 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:03:21.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-25T22:03:21.691 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:21.691 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:21.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:21.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:21.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:21.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:21.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:21.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:21.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:21.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:21.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:21.696 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:21.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:21.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:21.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:21.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:21.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:21.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:03:21.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-25T22:03:21.701 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:21.701 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.701 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.51242 2026-02-25T22:03:21.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.51242 2026-02-25T22:03:21.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:21.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:21.702 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:03:21.704 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:03:21.704 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:03:21.758 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2026-02-25T22:03:21.759 DEBUG:teuthology.orchestra.run.trial163:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-25T22:03:21.812 INFO:tasks.workunit.client.0.trial163.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:03:21.815 INFO:tasks.workunit.client.0.trial163.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:03:21.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: run_tests: export PATH=./bin:.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:03:21.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: run_tests: PATH=./bin:.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:03:21.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: run_tests: export CEPH_MON=127.0.0.1:7109 2026-02-25T22:03:21.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: run_tests: CEPH_MON=127.0.0.1:7109 2026-02-25T22:03:21.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: run_tests: export CEPH_ARGS 2026-02-25T22:03:21.816 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: run_tests: uuidgen 2026-02-25T22:03:21.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: run_tests: CEPH_ARGS+=' --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none ' 2026-02-25T22:03:21.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:21.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: run_tests: export CEPH_CONF=/dev/null 2026-02-25T22:03:21.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: run_tests: CEPH_CONF=/dev/null 2026-02-25T22:03:21.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: run_tests: set 2026-02-25T22:03:21.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:03:21.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: run_tests: local 'funcs=test_activate_osd 2026-02-25T22:03:21.820 INFO:tasks.workunit.client.0.trial163.stderr:test_activate_osd_after_mark_down 2026-02-25T22:03:21.820 INFO:tasks.workunit.client.0.trial163.stderr:test_activate_osd_skip_benchmark 2026-02-25T22:03:21.820 INFO:tasks.workunit.client.0.trial163.stderr:test_destroy_osd 2026-02-25T22:03:21.820 INFO:tasks.workunit.client.0.trial163.stderr:test_display_logs 2026-02-25T22:03:21.820 INFO:tasks.workunit.client.0.trial163.stderr:test_erasure_code_plugin_exists 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_expect_failure 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_file_with_random_data 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_flush_pg_stats 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_config 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_is_making_recovery_progress 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_last_scrub_stamp 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_not_primary 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_num_active_clean 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_num_active_or_peered 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_num_pgs 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_op_scheduler 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_osd_id_used_by_pgs 2026-02-25T22:03:21.821 INFO:tasks.workunit.client.0.trial163.stderr:test_get_osds 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_get_pg 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_get_primary 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_get_timeout_delays 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_is_clean 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_kill_daemon 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_kill_daemons 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_objectstore_tool 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_pg_schedule_scrub 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_pg_scrub 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_repair 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_run_in_background 2026-02-25T22:03:21.822 INFO:tasks.workunit.client.0.trial163.stderr:test_run_mon 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_run_osd 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_set_config 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_setup 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_teardown 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_background 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_for_clean 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_for_health_ok 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_for_osd 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_for_peered 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_for_scrub 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:test_wait_osd_id_used_by_pgs' 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: run_tests: local dir=td/ceph-helpers 2026-02-25T22:03:21.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_activate_osd td/ceph-helpers 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: test_activate_osd: local dir=td/ceph-helpers 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: test_activate_osd: setup td/ceph-helpers 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:21.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:21.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:21.827 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:21.827 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:21.827 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:21.827 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:21.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:21.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:21.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:21.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:21.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:21.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:21.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:21.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:21.834 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: test_activate_osd: run_mon td/ceph-helpers a 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:21.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:21.859 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:21.859 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:21.859 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:21.859 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:21.859 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.859 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:21.860 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:21.860 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:21.896 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:21.897 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:21.898 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:21.898 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:21.898 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:21.899 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:21.899 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:21.899 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:21.899 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:21.899 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:21.899 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.900 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:21.900 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:21.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:21.900 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:21.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:21.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:21.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:21.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:21.955 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:22.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: test_activate_osd: run_mgr td/ceph-helpers x 2026-02-25T22:03:22.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:22.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:22.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:22.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:22.008 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:22.009 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:22.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:22.119 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:22.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:884: test_activate_osd: run_osd td/ceph-helpers 0 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:22.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:22.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:22.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:22.143 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:22.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d1e9442-cf59-4eac-8b5d-394031414619 2026-02-25T22:03:22.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8d1e9442-cf59-4eac-8b5d-394031414619' 2026-02-25T22:03:22.144 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 8d1e9442-cf59-4eac-8b5d-394031414619 2026-02-25T22:03:22.144 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:22.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCqcZ9p9l4NCRAAe1ambwKXEzG90fyyNL53eQ== 2026-02-25T22:03:22.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCqcZ9p9l4NCRAAe1ambwKXEzG90fyyNL53eQ=="}' 2026-02-25T22:03:22.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d1e9442-cf59-4eac-8b5d-394031414619 -i td/ceph-helpers/0/new.json 2026-02-25T22:03:22.265 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:22.273 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:03:22.275 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqcZ9p9l4NCRAAe1ambwKXEzG90fyyNL53eQ== --osd-uuid 8d1e9442-cf59-4eac-8b5d-394031414619 2026-02-25T22:03:22.291 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:22.287+0000 7fd5390a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:22.293 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:22.287+0000 7fd5390a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:22.294 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:22.287+0000 7fd5390a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:22.295 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:22.291+0000 7fd5390a5a40 -1 bdev(0x55ea3b302400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:22.295 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:22.291+0000 7fd5390a5a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:03:23.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:03:23.106 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:23.106 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:23.107 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:23.107 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:23.217 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:23.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:23.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:23.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:23.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:23.219 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:23.231 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:23.227+0000 7ff6dbe92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:23.233 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:23.227+0000 7ff6dbe92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:23.234 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:23.227+0000 7ff6dbe92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:23.326 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:23.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:23.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:23.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:23.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:23.768 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:23.763+0000 7ff6dbe92a40 -1 Falling back to public interface 2026-02-25T22:03:24.433 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:24.433 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:24.434 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:24.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:24.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:24.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:24.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:24.739 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:24.735+0000 7ff6dbe92a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:25.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:25.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:25.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:25.542 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:25.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:25.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:25.752 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1952239823,v1:127.0.0.1:6801/1952239823] [v2:127.0.0.1:6802/1952239823,v1:127.0.0.1:6803/1952239823] exists,up 8d1e9442-cf59-4eac-8b5d-394031414619 2026-02-25T22:03:25.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:25.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:25.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:25.754 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: get_asok_path osd.0 2026-02-25T22:03:25.754 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:25.754 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:25.755 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:25.755 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:25.755 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:25.755 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:25.755 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: CEPH_ARGS= 2026-02-25T22:03:25.755 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:03:25.812 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-25T22:03:25.812 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2026-02-25T22:03:25.813 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: grep --quiet osd_max_backfills 2026-02-25T22:03:25.814 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:889: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:03:25.815 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:25.815 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:25.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:25.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:25.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:26.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:26.125 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:26.126 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:26.126 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:26.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:26.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:03:26.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-25T22:03:26.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:26.128 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:26.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:03:26.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-25T22:03:26.129 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:03:26.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:03:26.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:03:26.130 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:03:26.130 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:03:26.143 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:26.139+0000 7f3906c82a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:26.145 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:26.139+0000 7f3906c82a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:26.146 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:26.139+0000 7f3906c82a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:26.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:03:26.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:26.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:26.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:26.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:26.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:26.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:26.339 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:26.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:26.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:26.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:26.568 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:26.563+0000 7f3906c82a40 -1 Falling back to public interface 2026-02-25T22:03:27.225 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:27.219+0000 7f3906c82a40 -1 osd.0 6 log_to_monitors true 2026-02-25T22:03:27.559 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:27.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:27.560 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:27.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:27.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:27.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:27.771 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/2631180469,v1:127.0.0.1:6801/2631180469] [v2:127.0.0.1:6802/2631180469,v1:127.0.0.1:6803/2631180469] exists,up 8d1e9442-cf59-4eac-8b5d-394031414619 2026-02-25T22:03:27.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:27.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:27.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:27.773 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: get_asok_path osd.0 2026-02-25T22:03:27.773 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:27.774 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:27.774 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:27.774 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:27.774 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:27.774 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:27.774 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: CEPH_ARGS= 2026-02-25T22:03:27.775 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:894: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: teardown td/ceph-helpers 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:27.833 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:27.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:27.941 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:27.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:27.942 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:27.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:27.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:27.944 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:27.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:27.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:27.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:27.945 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:27.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:27.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:27.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:27.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:27.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:27.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:27.951 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:27.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd_after_mark_down: setup td/ceph-helpers 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:27.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:27.955 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:27.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:27.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:27.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:27.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:27.956 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:27.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:27.957 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:27.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:27.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:27.958 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:27.959 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:27.959 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:27.960 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:27.960 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:27.961 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:27.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:27.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:27.963 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:27.964 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:27.964 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:27.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:27.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:27.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:27.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:27.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:27.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:27.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:27.966 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:27.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:904: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:27.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:27.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:27.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:27.990 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:27.990 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:27.990 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:27.991 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:27.991 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:27.991 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:27.991 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:27.991 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:28.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:28.022 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:28.022 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:28.023 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:28.023 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:28.024 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:28.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:28.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:28.079 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:28.079 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:28.079 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:28.079 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:28.080 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:28.081 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:28.081 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:28.081 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:28.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:28.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:28.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:28.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:28.246 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:28.246 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:28.246 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:28.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:28.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:28.248 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:28.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:907: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2026-02-25T22:03:28.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:28.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:28.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:28.272 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:28.273 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a7f50565-f831-426d-9fe2-2c2bf2d0e893 2026-02-25T22:03:28.273 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a7f50565-f831-426d-9fe2-2c2bf2d0e893' 2026-02-25T22:03:28.273 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 a7f50565-f831-426d-9fe2-2c2bf2d0e893 2026-02-25T22:03:28.273 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:28.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwcZ9pOKi1EBAALujDEJip9c1G8xEfKWWJzQ== 2026-02-25T22:03:28.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwcZ9pOKi1EBAALujDEJip9c1G8xEfKWWJzQ=="}' 2026-02-25T22:03:28.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a7f50565-f831-426d-9fe2-2c2bf2d0e893 -i td/ceph-helpers/0/new.json 2026-02-25T22:03:28.394 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:28.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:03:28.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwcZ9pOKi1EBAALujDEJip9c1G8xEfKWWJzQ== --osd-uuid a7f50565-f831-426d-9fe2-2c2bf2d0e893 2026-02-25T22:03:28.419 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:28.415+0000 7fa8d06e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:28.421 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:28.415+0000 7fa8d06e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:28.422 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:28.415+0000 7fa8d06e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:28.422 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:28.419+0000 7fa8d06e7a40 -1 bdev(0x55abe7fc0400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:28.422 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:28.419+0000 7fa8d06e7a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:03:29.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:03:29.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:29.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:29.092 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:29.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:29.208 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:29.208 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:29.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:29.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:29.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:29.209 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:29.223 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:29.219+0000 7f3c07a2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:29.225 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:29.219+0000 7f3c07a2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:29.226 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:29.223+0000 7f3c07a2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:29.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:29.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:29.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:29.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:29.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:29.319 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:29.319 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:29.319 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:29.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:29.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:29.428 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:29.636 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:29.631+0000 7f3c07a2aa40 -1 Falling back to public interface 2026-02-25T22:03:30.308 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:30.303+0000 7f3c07a2aa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:30.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:30.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:30.430 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:30.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:30.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:30.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:30.545 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:31.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:31.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:31.547 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:31.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:31.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:31.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:31.756 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1817071345,v1:127.0.0.1:6801/1817071345] [v2:127.0.0.1:6802/1817071345,v1:127.0.0.1:6803/1817071345] exists,up a7f50565-f831-426d-9fe2-2c2bf2d0e893 2026-02-25T22:03:31.757 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:31.757 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:31.757 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:31.758 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-25T22:03:31.758 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:31.758 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:31.758 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:31.758 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:31.759 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:31.759 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:31.759 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-25T22:03:31.759 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:03:31.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2026-02-25T22:03:31.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2026-02-25T22:03:31.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2026-02-25T22:03:31.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:912: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:03:31.818 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:31.818 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:31.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:31.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:31.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:32.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:32.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: ceph osd down 0 2026-02-25T22:03:32.337 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 is already down. 2026-02-25T22:03:32.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: wait_for_osd down 0 2026-02-25T22:03:32.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-25T22:03:32.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:32.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:32.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:32.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:32.348 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:32.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:32.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:32.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-25T22:03:32.579 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/1817071345,v1:127.0.0.1:6801/1817071345] [v2:127.0.0.1:6802/1817071345,v1:127.0.0.1:6803/1817071345] exists a7f50565-f831-426d-9fe2-2c2bf2d0e893 2026-02-25T22:03:32.582 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:32.582 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:32.582 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:32.582 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:32.583 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:32.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-25T22:03:32.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:32.586 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:32.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:03:32.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-25T22:03:32.587 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:03:32.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:03:32.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:03:32.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:03:32.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:03:32.602 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:32.599+0000 7f5483016a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:32.604 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:32.599+0000 7f5483016a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:32.605 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:32.599+0000 7f5483016a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:32.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:03:32.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:32.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:32.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:32.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:32.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:32.804 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:32.804 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:32.804 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:32.804 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:33.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:33.020 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:33.015+0000 7f5483016a40 -1 Falling back to public interface 2026-02-25T22:03:33.701 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:33.695+0000 7f5483016a40 -1 osd.0 6 log_to_monitors true 2026-02-25T22:03:34.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:34.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:34.019 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:34.020 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:34.020 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:34.020 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:34.238 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6800/3315535748,v1:127.0.0.1:6801/3315535748] [v2:127.0.0.1:6802/3315535748,v1:127.0.0.1:6803/3315535748] exists,up a7f50565-f831-426d-9fe2-2c2bf2d0e893 2026-02-25T22:03:34.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:34.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:34.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:34.239 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-25T22:03:34.239 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:34.239 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:34.240 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:34.240 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.240 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.240 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:34.241 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-25T22:03:34.241 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:03:34.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2026-02-25T22:03:34.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-25T22:03:34.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2026-02-25T22:03:34.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:34.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:34.303 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:34.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:34.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:34.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:34.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:34.305 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:34.409 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:34.409 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:34.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:34.410 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:34.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:34.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:34.412 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:34.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:34.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:34.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:34.414 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:34.415 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:34.416 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:34.416 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:34.420 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:34.420 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.420 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.420 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:34.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:34.424 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:34.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:34.425 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:34.426 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:34.426 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:34.427 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:34.427 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:34.427 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:34.428 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:34.428 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:34.429 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:34.429 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:34.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:34.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:34.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:34.432 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:34.433 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.433 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.433 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:34.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:34.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:34.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:34.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:34.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.436 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:34.437 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:34.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:34.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:34.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:34.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2026-02-25T22:03:34.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:34.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:34.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:34.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:34.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:34.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:34.459 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:34.459 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:34.459 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:34.460 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:34.460 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.460 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.460 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:34.460 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:34.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:34.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:34.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:34.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:34.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:34.491 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:34.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:34.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:34.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:34.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:34.549 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:34.549 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:34.550 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:34.551 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:34.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2026-02-25T22:03:34.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:34.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:34.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:34.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:34.605 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:34.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:34.717 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:34.718 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:34.718 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:34.718 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:34.718 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.718 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.718 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:34.719 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:34.719 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:34.741 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:34.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:34.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-25T22:03:34.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:34.745 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:34.745 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=99fee95d-4de1-4dc1-9121-2ea195b412b3 2026-02-25T22:03:34.745 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 99fee95d-4de1-4dc1-9121-2ea195b412b3' 2026-02-25T22:03:34.746 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 99fee95d-4de1-4dc1-9121-2ea195b412b3 2026-02-25T22:03:34.746 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:34.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2cZ9pxwD3LBAAWMmFOl66mcIbODG/B8tzBQ== 2026-02-25T22:03:34.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2cZ9pxwD3LBAAWMmFOl66mcIbODG/B8tzBQ=="}' 2026-02-25T22:03:34.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 99fee95d-4de1-4dc1-9121-2ea195b412b3 -i td/ceph-helpers/0/new.json 2026-02-25T22:03:34.864 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:34.872 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:03:34.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQC2cZ9pxwD3LBAAWMmFOl66mcIbODG/B8tzBQ== --osd-uuid 99fee95d-4de1-4dc1-9121-2ea195b412b3 2026-02-25T22:03:34.889 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:34.883+0000 7f01670a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:34.890 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:34.887+0000 7f01670a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:34.891 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:34.887+0000 7f01670a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:34.892 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:34.887+0000 7f01670a1a40 -1 bdev(0x5621b7214400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:34.892 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:34.887+0000 7f01670a1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:03:35.645 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:03:35.645 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:35.646 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:35.646 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:35.647 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:35.768 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:35.768 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:35.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:03:35.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:35.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:35.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:35.791 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:35.787+0000 7fc335eeba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:35.793 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:35.787+0000 7fc335eeba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:35.795 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:35.791+0000 7fc335eeba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:35.880 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:35.880 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:35.880 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:35.880 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:35.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:35.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:35.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:35.881 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:35.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:35.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:35.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:36.232 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:36.227+0000 7fc335eeba40 -1 Falling back to public interface 2026-02-25T22:03:36.881 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:36.875+0000 7fc335eeba40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:36.991 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:36.991 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:36.991 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:36.992 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:36.992 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:36.992 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:37.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:38.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:38.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:38.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:38.104 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:38.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:38.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:38.315 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1968023663,v1:127.0.0.1:6801/1968023663] [v2:127.0.0.1:6802/1968023663,v1:127.0.0.1:6803/1968023663] exists,up 99fee95d-4de1-4dc1-9121-2ea195b412b3 2026-02-25T22:03:38.316 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:38.316 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:38.316 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:38.317 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:03:38.317 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:03:38.318 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-25T22:03:38.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-25T22:03:38.378 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:03:38.378 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:38.378 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:38.379 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:38.379 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:38.379 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:38.379 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:38.379 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:03:38.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-25T22:03:38.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-25T22:03:38.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:940: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:03:38.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:38.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:38.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:38.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:38.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:38.748 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:38.748 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:941: test_activate_osd_skip_benchmark: ceph osd down 0 2026-02-25T22:03:38.961 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 is already down. 2026-02-25T22:03:38.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: wait_for_osd down 0 2026-02-25T22:03:38.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-25T22:03:38.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:38.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:38.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:38.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:38.972 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:38.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:38.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:38.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-25T22:03:39.184 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/1968023663,v1:127.0.0.1:6801/1968023663] [v2:127.0.0.1:6802/1968023663,v1:127.0.0.1:6803/1968023663] exists 99fee95d-4de1-4dc1-9121-2ea195b412b3 2026-02-25T22:03:39.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:39.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:39.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:946: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:39.188 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:39.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:39.190 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:39.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:03:39.192 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:03:39.192 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:03:39.192 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:03:39.192 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:03:39.192 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:03:39.192 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:03:39.207 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:39.203+0000 7fe050ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:39.209 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:39.203+0000 7fe050ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:39.209 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:39.203+0000 7fe050ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:39.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:03:39.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:39.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:39.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:39.399 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:39.400 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:39.400 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:39.400 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:39.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:39.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:39.583 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:39.579+0000 7fe050ed0a40 -1 Falling back to public interface 2026-02-25T22:03:39.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:40.186 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:40.183+0000 7fe050ed0a40 -1 osd.0 5 log_to_monitors true 2026-02-25T22:03:40.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:40.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:40.611 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:40.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:40.611 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:40.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:40.830 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/1239844842,v1:127.0.0.1:6801/1239844842] [v2:127.0.0.1:6802/1239844842,v1:127.0.0.1:6803/1239844842] exists,up 99fee95d-4de1-4dc1-9121-2ea195b412b3 2026-02-25T22:03:40.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:40.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:40.831 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:40.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:03:40.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:40.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:40.832 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:40.832 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:40.832 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:40.833 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:40.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:03:40.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-25T22:03:40.891 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-25T22:03:40.892 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:03:40.893 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:40.893 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:40.894 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:40.894 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:40.894 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:40.894 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:40.894 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:03:40.894 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:953: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:954: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:40.954 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:40.955 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:40.955 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:40.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:40.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:40.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:41.062 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:41.062 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:41.063 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:41.063 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:41.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:41.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:41.065 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:41.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:41.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:41.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:41.067 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:41.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:41.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:41.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:41.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:41.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.074 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.074 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:41.076 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:41.076 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_destroy_osd td/ceph-helpers 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: test_destroy_osd: local dir=td/ceph-helpers 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: test_destroy_osd: setup td/ceph-helpers 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:41.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:41.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:41.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:41.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:41.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:41.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:41.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:41.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:41.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:41.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:41.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:41.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:41.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:41.083 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:41.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:41.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:41.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:41.087 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:41.087 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.087 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:41.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:41.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:41.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:41.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:41.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:41.091 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: test_destroy_osd: run_mon td/ceph-helpers a 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:41.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:41.113 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:41.113 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:41.113 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:41.113 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:41.113 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.113 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.114 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:41.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:41.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:41.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:41.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:41.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:41.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:41.143 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.144 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:41.145 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:41.145 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:41.198 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:41.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:41.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:41.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:41.199 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:41.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:41.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:41.200 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:41.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:41.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.200 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.201 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:41.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:41.201 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:41.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: test_destroy_osd: run_mgr td/ceph-helpers x 2026-02-25T22:03:41.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:41.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:41.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:41.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:41.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:41.254 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:41.367 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:41.367 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:41.367 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:41.368 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:41.368 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.368 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.368 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:41.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:41.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:41.388 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: test_destroy_osd: run_osd td/ceph-helpers 0 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:41.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:41.390 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:41.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:41.392 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:41.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3615a6d5-4041-4306-8b8d-845402c16ed0 2026-02-25T22:03:41.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3615a6d5-4041-4306-8b8d-845402c16ed0' 2026-02-25T22:03:41.392 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 3615a6d5-4041-4306-8b8d-845402c16ed0 2026-02-25T22:03:41.393 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:41.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC9cZ9pcNrlFxAAfzbN9OBNjwQ46W+oLpyUlQ== 2026-02-25T22:03:41.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC9cZ9pcNrlFxAAfzbN9OBNjwQ46W+oLpyUlQ=="}' 2026-02-25T22:03:41.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3615a6d5-4041-4306-8b8d-845402c16ed0 -i td/ceph-helpers/0/new.json 2026-02-25T22:03:41.515 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:41.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:03:41.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9cZ9pcNrlFxAAfzbN9OBNjwQ46W+oLpyUlQ== --osd-uuid 3615a6d5-4041-4306-8b8d-845402c16ed0 2026-02-25T22:03:41.540 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:41.534+0000 7fdb329eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:41.542 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:41.534+0000 7fdb329eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:41.543 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:41.538+0000 7fdb329eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:41.543 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:41.538+0000 7fdb329eaa40 -1 bdev(0x558fb39a4400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:41.543 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:41.538+0000 7fdb329eaa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:03:42.321 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:03:42.322 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:42.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:42.323 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:42.323 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:42.433 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:42.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:42.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:42.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:42.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:42.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:42.448 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:42.442+0000 7fbbd4bc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:42.449 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:42.442+0000 7fbbd4bc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:42.450 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:42.442+0000 7fbbd4bc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:42.545 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:42.545 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:42.545 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:42.545 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:42.545 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:42.546 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:42.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:42.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:42.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:42.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:42.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:42.960 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:42.954+0000 7fbbd4bc4a40 -1 Falling back to public interface 2026-02-25T22:03:43.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:43.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:43.651 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:43.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:43.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:43.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:43.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:43.924 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:43.918+0000 7fbbd4bc4a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:44.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:44.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:44.765 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:44.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:44.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:44.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:44.970 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2719110479,v1:127.0.0.1:6801/2719110479] [v2:127.0.0.1:6802/2719110479,v1:127.0.0.1:6803/2719110479] exists,up 3615a6d5-4041-4306-8b8d-845402c16ed0 2026-02-25T22:03:44.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:44.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:44.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:44.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:793: test_destroy_osd: destroy_osd td/ceph-helpers 0 2026-02-25T22:03:44.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:775: destroy_osd: local dir=td/ceph-helpers 2026-02-25T22:03:44.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:776: destroy_osd: local id=0 2026-02-25T22:03:44.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:778: destroy_osd: ceph osd out osd.0 2026-02-25T22:03:45.240 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 is already out. 2026-02-25T22:03:45.249 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:779: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-25T22:03:45.250 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:45.250 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:45.250 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:45.250 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:45.250 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:45.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:45.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:780: destroy_osd: ceph osd down osd.0 2026-02-25T22:03:45.773 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 is already down. 2026-02-25T22:03:45.783 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:781: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2026-02-25T22:03:46.003 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 does not exist 2026-02-25T22:03:46.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: teardown td/ceph-helpers/0 2026-02-25T22:03:46.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers/0 2026-02-25T22:03:46.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:46.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers/0 KILL 2026-02-25T22:03:46.015 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:46.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:46.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:46.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:46.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:46.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:46.018 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:46.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:46.019 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:46.020 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:46.020 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:46.020 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:46.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:46.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:46.022 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.023 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:46.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:46.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers/0 2026-02-25T22:03:46.027 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:46.027 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.027 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.027 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:46.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:46.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:46.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: rm -fr td/ceph-helpers/0 2026-02-25T22:03:46.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: ceph osd dump 2026-02-25T22:03:46.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: grep 'osd. ' 2026-02-25T22:03:46.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:795: test_destroy_osd: teardown td/ceph-helpers 2026-02-25T22:03:46.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:46.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:46.247 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:46.248 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:46.248 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:46.248 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:46.248 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:46.248 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:46.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:46.355 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:46.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:46.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:46.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:46.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:46.358 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:46.359 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.359 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:46.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:46.360 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:46.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:46.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:46.365 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:46.365 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.365 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.365 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_display_logs td/ceph-helpers 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_display_logs: local dir=td/ceph-helpers 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_display_logs: setup td/ceph-helpers 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:46.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:46.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:46.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:46.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:46.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:46.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:46.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:46.370 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:46.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:46.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:46.372 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:46.372 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:46.372 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:46.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:46.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:46.374 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:46.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:46.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:46.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:46.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:46.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:46.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:46.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:46.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:46.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:46.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_display_logs: run_mon td/ceph-helpers a 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:46.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:46.405 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:46.405 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:46.405 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:46.405 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:46.405 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.405 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.406 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:46.406 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:46.433 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:46.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:46.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:46.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:46.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:46.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:46.436 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:46.436 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:46.436 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:46.436 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:46.437 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.437 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.437 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:46.437 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:46.437 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:46.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:46.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:46.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:46.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.494 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.495 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:46.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:46.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:46.550 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_display_logs: kill_daemons td/ceph-helpers 2026-02-25T22:03:46.551 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:46.551 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:46.551 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:46.552 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:46.552 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:46.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:46.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_display_logs: display_logs td/ceph-helpers 2026-02-25T22:03:46.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: display_logs: local dir=td/ceph-helpers 2026-02-25T22:03:46.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2026-02-25T22:03:46.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: display_logs: read file 2026-02-25T22:03:46.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2026-02-25T22:03:46.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: display_logs: cat td/ceph-helpers/mon.a.log 2026-02-25T22:03:46.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: display_logs: read file 2026-02-25T22:03:46.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2164: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2026-02-25T22:03:46.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_display_logs: teardown td/ceph-helpers 2026-02-25T22:03:46.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:46.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:46.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:46.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:46.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:46.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:46.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:46.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:46.665 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:46.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:46.666 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:46.666 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:46.667 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:46.667 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:46.667 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:46.668 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.668 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:46.669 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:46.669 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.670 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:46.671 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:46.671 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:46.672 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:46.673 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.673 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.673 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:46.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:46.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:46.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:46.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: test_erasure_code_plugin_exists: setup td/ceph-helpers 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:46.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:46.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:46.677 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:46.678 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:46.678 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:46.680 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:46.680 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:46.680 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:46.681 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.681 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:46.682 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:46.682 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:46.683 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:46.684 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:46.684 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:46.685 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:46.685 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.685 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.685 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:46.686 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:46.686 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:46.686 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:46.687 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:46.687 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.688 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:46.689 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:46.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:46.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:46.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:46.690 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2026-02-25T22:03:46.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:46.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:46.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:46.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:46.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:46.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:46.710 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:46.710 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:46.710 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:46.710 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:46.710 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.710 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.711 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:46.711 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:46.738 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:46.740 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:46.740 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:46.740 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:46.740 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:46.740 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:46.741 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.741 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.741 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:46.741 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:46.741 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:46.795 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:46.796 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:46.796 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:46.796 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.797 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:46.798 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:46.798 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:46.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:46.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:46.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:46.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:46.967 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:46.967 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:46.967 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:46.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:46.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:46.968 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:46.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2026-02-25T22:03:46.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=jerasure 2026-02-25T22:03:46.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-25T22:03:46.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-25T22:03:46.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-25T22:03:46.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-25T22:03:46.987 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-25T22:03:46.987 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2026-02-25T22:03:46.987 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2026-02-25T22:03:47.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s= 2026-02-25T22:03:47.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=0 2026-02-25T22:03:47.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-25T22:03:47.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-25T22:03:47.261 INFO:tasks.workunit.client.0.trial163.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-25T22:03:47.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 0 2026-02-25T22:03:47.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2026-02-25T22:03:47.269 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=FAKE 2026-02-25T22:03:47.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-25T22:03:47.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-25T22:03:47.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-25T22:03:47.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-25T22:03:47.270 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-25T22:03:47.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2026-02-25T22:03:47.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2026-02-25T22:03:47.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2026-02-25T22:03:47.367 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=5 2026-02-25T22:03:47.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2026-02-25T22:03:47.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2026-02-25T22:03:47.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2119: erasure_code_plugin_exists: status=1 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 1 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:47.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:47.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:47.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:47.477 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:47.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:47.478 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:47.479 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:47.479 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:47.480 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:47.481 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.481 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:47.481 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:47.481 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:47.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:47.484 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:47.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:47.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:47.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:47.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:47.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:47.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_expect_failure td/ceph-helpers 2026-02-25T22:03:47.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_expect_failure: local dir=td/ceph-helpers 2026-02-25T22:03:47.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_expect_failure: setup td/ceph-helpers 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:47.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:47.490 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:47.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:47.491 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:47.491 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:47.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:47.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:47.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:47.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:47.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:47.495 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.496 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:47.496 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:47.497 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:47.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:47.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.498 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:47.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:47.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:47.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:47.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:47.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.501 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:47.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:47.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:47.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:47.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:47.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2026-02-25T22:03:47.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-25T22:03:47.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:47.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-25T22:03:47.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:47.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:47.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo FAIL ; exit 1' 2026-02-25T22:03:47.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:03:47.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:03:47.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2042: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:47.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo FAIL ; exit 0' 2026-02-25T22:03:47.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: expect_failure: success=true 2026-02-25T22:03:47.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: true 2026-02-25T22:03:47.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: cat td/ceph-helpers/out 2026-02-25T22:03:47.508 INFO:tasks.workunit.client.0.trial163.stderr:FAIL 2026-02-25T22:03:47.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: expect_failure: return 1 2026-02-25T22:03:47.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2043: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:03:47.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2026-02-25T22:03:47.511 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:03:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:03:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:03:47.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: cat td/ceph-helpers/out 2026-02-25T22:03:47.513 INFO:tasks.workunit.client.0.trial163.stderr:UNEXPECTED 2026-02-25T22:03:47.514 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: expect_failure: return 1 2026-02-25T22:03:47.514 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:03:47.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_expect_failure: teardown td/ceph-helpers 2026-02-25T22:03:47.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:47.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:47.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:47.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:47.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:47.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:47.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:47.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:47.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:47.518 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:47.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:47.519 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:47.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:47.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:47.521 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:47.522 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.522 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:47.522 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:47.522 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:47.524 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:47.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:47.526 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:47.526 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.526 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:47.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:47.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:47.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_file_with_random_data td/ceph-helpers 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: test_file_with_random_data: local dir=td/ceph-helpers 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: test_file_with_random_data: setup td/ceph-helpers 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:47.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:47.530 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:47.530 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:47.531 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:47.531 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:47.532 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:47.532 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:47.533 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:47.533 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:47.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:47.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:47.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:47.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:47.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:47.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:47.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:47.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:47.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:47.539 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:47.540 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:47.540 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.540 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.540 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:47.542 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:47.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:47.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:47.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:47.543 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: test_file_with_random_data: file_with_random_data 4000 2026-02-25T22:03:47.543 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2343: file_with_random_data: local size_bytes=4000 2026-02-25T22:03:47.543 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: mktemp 2026-02-25T22:03:47.544 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local file=/tmp/tmp.SwFY7b50EP 2026-02-25T22:03:47.544 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.SwFY7b50EP bs=4000 count=1 2026-02-25T22:03:47.545 INFO:tasks.workunit.client.0.trial163.stderr:1+0 records in 2026-02-25T22:03:47.545 INFO:tasks.workunit.client.0.trial163.stderr:1+0 records out 2026-02-25T22:03:47.545 INFO:tasks.workunit.client.0.trial163.stderr:4000 bytes (4.0 kB, 3.9 KiB) copied, 5.6866e-05 s, 70.3 MB/s 2026-02-25T22:03:47.545 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: printf %s /tmp/tmp.SwFY7b50EP 2026-02-25T22:03:47.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: test_file_with_random_data: local file=/tmp/tmp.SwFY7b50EP 2026-02-25T22:03:47.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: test_file_with_random_data: test -f /tmp/tmp.SwFY7b50EP 2026-02-25T22:03:47.546 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_file_with_random_data: stat -c %s /tmp/tmp.SwFY7b50EP 2026-02-25T22:03:47.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_file_with_random_data: test 4000 = 4000 2026-02-25T22:03:47.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: test_file_with_random_data: rm /tmp/tmp.SwFY7b50EP 2026-02-25T22:03:47.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: test_file_with_random_data: teardown td/ceph-helpers 2026-02-25T22:03:47.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:47.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:47.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:47.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:47.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:47.549 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:47.549 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:47.549 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:47.550 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:47.551 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:47.551 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:47.552 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:47.553 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:47.553 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:47.553 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:47.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:47.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:47.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.556 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:47.557 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:47.557 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:47.558 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:47.558 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.558 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_flush_pg_stats td/ceph-helpers 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: test_flush_pg_stats: local dir=td/ceph-helpers 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: test_flush_pg_stats: setup td/ceph-helpers 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:47.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:47.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:47.562 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:47.562 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:47.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:47.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:47.562 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:47.563 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:47.563 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:47.564 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:47.564 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:47.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:47.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:47.565 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:47.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:47.567 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:47.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:47.568 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:47.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:47.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:47.570 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:47.570 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.570 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.570 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:47.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:47.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:47.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:47.573 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:47.573 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.573 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.573 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:47.575 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:47.576 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:47.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:47.577 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:03:47.598 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:47.598 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:47.598 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:47.598 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:47.598 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.598 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.599 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:47.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:03:47.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:47.634 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:47.634 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:47.635 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:47.635 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.636 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:47.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:47.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:47.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:47.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:47.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:47.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:47.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:47.694 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:47.694 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:47.694 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:47.695 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:47.695 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.695 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.695 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:47.695 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:47.695 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_flush_pg_stats: run_mgr td/ceph-helpers x 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:47.750 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:47.862 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:47.863 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:47.863 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:47.863 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:47.863 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.863 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.864 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:47.864 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:47.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:47.883 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: test_flush_pg_stats: run_osd td/ceph-helpers 0 2026-02-25T22:03:47.883 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:47.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:47.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:03:47.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:47.887 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:47.887 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=750c2043-a90e-4a18-b0d0-04b17719e4fb 2026-02-25T22:03:47.887 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 750c2043-a90e-4a18-b0d0-04b17719e4fb 2026-02-25T22:03:47.888 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 750c2043-a90e-4a18-b0d0-04b17719e4fb' 2026-02-25T22:03:47.888 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:47.896 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDDcZ9pJopeNRAA6bKBDk8vSC9xhuyjJ8Qsgw== 2026-02-25T22:03:47.896 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDDcZ9pJopeNRAA6bKBDk8vSC9xhuyjJ8Qsgw=="}' 2026-02-25T22:03:47.896 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 750c2043-a90e-4a18-b0d0-04b17719e4fb -i td/ceph-helpers/0/new.json 2026-02-25T22:03:48.005 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:48.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:03:48.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDcZ9pJopeNRAA6bKBDk8vSC9xhuyjJ8Qsgw== --osd-uuid 750c2043-a90e-4a18-b0d0-04b17719e4fb 2026-02-25T22:03:48.029 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.022+0000 7fc2aaab4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:48.030 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.026+0000 7fc2aaab4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:48.031 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.026+0000 7fc2aaab4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:48.032 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.026+0000 7fc2aaab4a40 -1 bdev(0x5564add1c400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:48.032 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.026+0000 7fc2aaab4a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:03:48.674 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:03:48.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:48.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:48.676 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:48.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:48.794 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:48.794 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:48.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:03:48.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:48.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:48.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:48.820 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.814+0000 7fa0ea980a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:48.821 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.814+0000 7fa0ea980a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:48.822 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:48.818+0000 7fa0ea980a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:48.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:48.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:48.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:48.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:48.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:48.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:48.907 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:48.907 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:48.907 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:48.907 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:49.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:49.180 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:49.174+0000 7fa0ea980a40 -1 Falling back to public interface 2026-02-25T22:03:50.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:50.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:50.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:50.018 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:50.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:50.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:50.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:50.239 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:50.234+0000 7fa0ea980a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:51.156 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:51.156 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:51.156 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:51.157 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:51.157 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:51.157 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:51.374 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3746961854,v1:127.0.0.1:6801/3746961854] [v2:127.0.0.1:6802/3746961854,v1:127.0.0.1:6803/3746961854] exists,up 750c2043-a90e-4a18-b0d0-04b17719e4fb 2026-02-25T22:03:51.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:51.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:51.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:51.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: test_flush_pg_stats: create_rbd_pool 2026-02-25T22:03:51.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:03:51.581 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:03:51.590 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:03:51.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:03:51.817 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:03:51.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:03:52.830 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:03:53.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_flush_pg_stats: rados -p rbd put obj /etc/group 2026-02-25T22:03:53.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_flush_pg_stats: flush_pg_stats 2026-02-25T22:03:53.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:03:53.127 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:03:53.352 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:03:53.352 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:03:53.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:03:53.353 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:03:53.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:03:53.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:03:53.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:03:53.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:03:53.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:03:53.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:03:53.436 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:03:53.437 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:03:53.437 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:03:53.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:03:53.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:03:53.439 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:03:53.439 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:03:53.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:03:53.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:03:54.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:03:54.650 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:03:54.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:03:54.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2026-02-25T22:03:54.874 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: ceph df detail --format=json 2026-02-25T22:03:54.874 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2026-02-25T22:03:55.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: stored=8398 2026-02-25T22:03:55.161 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: ceph df detail --format=json 2026-02-25T22:03:55.161 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: stored_raw=8398 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: test_flush_pg_stats: test 8398 -gt 0 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2293: test_flush_pg_stats: test 8398 == 8398 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: test_flush_pg_stats: teardown td/ceph-helpers 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:55.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:55.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:55.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:55.749 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:55.750 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:55.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:55.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:55.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:55.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:55.753 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:55.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:55.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:55.755 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:55.755 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:55.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:55.757 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:55.757 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:55.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:55.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_config td/ceph-helpers 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: test_get_config: local dir=td/ceph-helpers 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: test_get_config: setup td/ceph-helpers 2026-02-25T22:03:55.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:55.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:55.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:55.767 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:55.768 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:55.768 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:55.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:55.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:55.769 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:55.770 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:55.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:55.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:55.771 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:55.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:55.773 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:55.773 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:55.774 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:55.774 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.775 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.775 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:55.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:55.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:55.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:55.777 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:55.778 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.778 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:55.779 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:55.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:55.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:55.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:55.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1131: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:03:55.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:55.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:55.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:55.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:55.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:55.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.802 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:55.803 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:03:55.829 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:55.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:55.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:55.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:55.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:55.831 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.832 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:55.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:55.833 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:03:55.895 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:03:55.895 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:55.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:55.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.897 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.898 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:55.898 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:55.898 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:03:55.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: get_config mon a osd_pool_default_size 2026-02-25T22:03:55.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:55.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:55.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_pool_default_size 2026-02-25T22:03:55.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_pool_default_size 2026-02-25T22:03:55.953 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:55.953 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:55.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get osd_pool_default_size 2026-02-25T22:03:56.010 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: test 1 = 1 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: test_get_config: run_mgr td/ceph-helpers x 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:03:56.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:03:56.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:03:56.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:56.126 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:56.126 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:56.126 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:56.126 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:56.126 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:56.127 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:56.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:03:56.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2026-02-25T22:03:56.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:03:56.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:03:56.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:03:56.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:03:56.149 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_max_scrubs=3 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:03:56.150 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:03:56.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dac23df1-3a01-4e0f-ada6-016ca3f745af 2026-02-25T22:03:56.151 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 dac23df1-3a01-4e0f-ada6-016ca3f745af 2026-02-25T22:03:56.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dac23df1-3a01-4e0f-ada6-016ca3f745af' 2026-02-25T22:03:56.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:03:56.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMcZ9pzXuDCRAA5S564X5asGBD4TgJrbnrfw== 2026-02-25T22:03:56.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMcZ9pzXuDCRAA5S564X5asGBD4TgJrbnrfw=="}' 2026-02-25T22:03:56.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dac23df1-3a01-4e0f-ada6-016ca3f745af -i td/ceph-helpers/0/new.json 2026-02-25T22:03:56.275 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:56.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:03:56.285 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQDMcZ9pzXuDCRAA5S564X5asGBD4TgJrbnrfw== --osd-uuid dac23df1-3a01-4e0f-ada6-016ca3f745af 2026-02-25T22:03:56.301 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:56.294+0000 7f343989ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:56.303 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:56.298+0000 7f343989ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:56.303 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:56.298+0000 7f343989ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:56.304 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:56.298+0000 7f343989ea40 -1 bdev(0x55d37a6ce400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:03:56.304 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:56.298+0000 7f343989ea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:03:57.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:03:57.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:03:57.034 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:03:57.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:03:57.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:03:57.154 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:03:57.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:03:57.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2026-02-25T22:03:57.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:03:57.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:03:57.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:03:57.180 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:57.174+0000 7fbf26b95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:57.182 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:57.174+0000 7fbf26b95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:57.183 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:57.178+0000 7fbf26b95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:03:57.262 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:03:57.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:03:57.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:03:57.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:03:57.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:03:57.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:57.263 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:03:57.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:03:57.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:57.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:57.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:57.524 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:57.518+0000 7fbf26b95a40 -1 Falling back to public interface 2026-02-25T22:03:58.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:58.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:58.380 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:03:58.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:03:58.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:58.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:58.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:03:58.595 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:03:58.590+0000 7fbf26b95a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:03:59.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:03:59.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:03:59.488 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:03:59.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:03:59.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:03:59.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:03:59.697 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2960102304,v1:127.0.0.1:6801/2960102304] [v2:127.0.0.1:6802/2960102304,v1:127.0.0.1:6803/2960102304] exists,up dac23df1-3a01-4e0f-ada6-016ca3f745af 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: get_config osd 0 osd_max_scrubs 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_max_scrubs 2026-02-25T22:03:59.698 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_max_scrubs 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:59.700 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_max_scrubs 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: test 3 = 3 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: teardown td/ceph-helpers 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:59.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:59.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:59.871 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:59.871 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:59.871 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:59.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:59.873 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:59.873 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:59.874 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:59.875 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:59.875 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:59.875 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:59.876 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:59.877 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:59.877 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:59.881 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:59.881 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:59.881 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:59.881 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: test_get_is_making_recovery_progress: setup td/ceph-helpers 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:03:59.884 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:03:59.885 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:03:59.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:03:59.886 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:03:59.887 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:03:59.887 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:03:59.888 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:03:59.888 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:03:59.889 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:03:59.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:59.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:03:59.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:03:59.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:03:59.891 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:03:59.892 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:03:59.892 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:03:59.894 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:03:59.894 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:59.894 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:59.894 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:03:59.895 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:03:59.895 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:03:59.895 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:03:59.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:03:59.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:59.896 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:59.896 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:03:59.898 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:03:59.899 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:03:59.899 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:03:59.899 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:03:59.899 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2026-02-25T22:03:59.899 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:03:59.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:03:59.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:03:59.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:03:59.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:03:59.900 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:03:59.921 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:03:59.921 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:03:59.921 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:03:59.921 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:03:59.921 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:59.921 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:59.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:03:59.922 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:03:59.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:03:59.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:03:59.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:03:59.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:03:59.951 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:03:59.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:03:59.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:00.010 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:00.010 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:00.010 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:00.010 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:00.010 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:00.012 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:00.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2026-02-25T22:04:00.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:00.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:00.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:00.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:00.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:00.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:00.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:00.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:00.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:00.185 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:00.185 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.185 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.185 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:00.185 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:00.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:00.202 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2026-02-25T22:04:00.202 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:04:00.203 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:04:00.203 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:04:00.203 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:04:00.203 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:04:00.203 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1341: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:00.315 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:00.316 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:00.316 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:00.422 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:00.422 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:00.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:00.424 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:00.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:00.425 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:00.426 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:00.427 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:00.427 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:00.427 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:00.427 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:00.428 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:00.429 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:00.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:00.431 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:00.432 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.432 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:00.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:00.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:00.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:00.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1524: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: test_get_last_scrub_stamp: setup td/ceph-helpers 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:00.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:00.436 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:00.436 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:00.436 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:00.437 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:00.437 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:00.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:00.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:00.439 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:00.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:00.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:00.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:00.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:00.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:00.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:00.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:00.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:00.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.444 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:00.445 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:00.445 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:00.445 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:00.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:00.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.447 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.447 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:00.448 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:00.449 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:00.449 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:00.449 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:00.449 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:00.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:00.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:00.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:00.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:00.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:00.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:00.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:00.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:00.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:00.472 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:00.472 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.472 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.473 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:00.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:00.501 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:00.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:00.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:00.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:00.502 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:00.503 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:00.504 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:00.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:00.561 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:00.561 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:00.561 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:00.561 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:00.561 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:00.562 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:00.562 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:00.562 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:00.563 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:00.563 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.563 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.563 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:00.563 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:00.563 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:00.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:00.741 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:00.741 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:00.741 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:00.742 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:00.742 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.742 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:00.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:00.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:00.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2026-02-25T22:04:00.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:00.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:00.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:00.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:00.765 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:00.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:00.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:00.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:00.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:00.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:00.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:00.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:00.768 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:00.768 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=432b24be-53dc-448b-af56-1cecc0f64ae4 2026-02-25T22:04:00.768 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 432b24be-53dc-448b-af56-1cecc0f64ae4' 2026-02-25T22:04:00.768 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 432b24be-53dc-448b-af56-1cecc0f64ae4 2026-02-25T22:04:00.768 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:00.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDQcZ9pMlM5LhAAGa/1tz3QZ5LjDUspvMBXTw== 2026-02-25T22:04:00.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDQcZ9pMlM5LhAAGa/1tz3QZ5LjDUspvMBXTw=="}' 2026-02-25T22:04:00.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 432b24be-53dc-448b-af56-1cecc0f64ae4 -i td/ceph-helpers/0/new.json 2026-02-25T22:04:00.888 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:00.897 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:00.898 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQcZ9pMlM5LhAAGa/1tz3QZ5LjDUspvMBXTw== --osd-uuid 432b24be-53dc-448b-af56-1cecc0f64ae4 2026-02-25T22:04:00.913 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:00.906+0000 7fc6847c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:00.914 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:00.910+0000 7fc6847c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:00.916 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:00.910+0000 7fc6847c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:00.916 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:00.910+0000 7fc6847c2a40 -1 bdev(0x563987aa4400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:00.917 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:00.910+0000 7fc6847c2a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:01.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:01.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:01.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:01.700 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:01.701 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:01.827 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:01.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:01.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:01.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:01.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:01.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:01.851 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:01.846+0000 7f32a20e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:01.854 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:01.850+0000 7f32a20e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:01.855 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:01.850+0000 7f32a20e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:01.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:01.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:01.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:01.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:01.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:01.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:01.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:01.942 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:01.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:01.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:02.054 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:02.288 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:02.282+0000 7f32a20e2a40 -1 Falling back to public interface 2026-02-25T22:04:03.043 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:03.038+0000 7f32a20e2a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:04:03.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:03.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:03.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:03.055 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:03.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:03.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:03.165 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:04.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:04.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:04.168 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:04.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:04.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:04.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:04.379 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2660120765,v1:127.0.0.1:6801/2660120765] [v2:127.0.0.1:6802/2660120765,v1:127.0.0.1:6803/2660120765] exists,up 432b24be-53dc-448b-af56-1cecc0f64ae4 2026-02-25T22:04:04.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:04.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:04.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:04.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: test_get_last_scrub_stamp: create_rbd_pool 2026-02-25T22:04:04.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:04:04.589 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:04:04.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:04:04.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:04:04.841 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:04:04.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:04:05.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:04:06.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: test_get_last_scrub_stamp: wait_for_clean 2026-02-25T22:04:06.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:04:06.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:04:06.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:04:06.127 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:04:06.127 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:04:06.128 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:04:06.128 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:04:06.128 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:04:06.128 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:04:06.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:04:06.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:04:06.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:04:06.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:04:06.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:04:06.227 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:04:06.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:04:06.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:04:06.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:04:06.439 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:04:06.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:04:06.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:04:06.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:04:06.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:04:06.521 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:06.521 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:04:06.521 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:04:06.522 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:06.522 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:04:06.524 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:04:06.524 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:04:06.524 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:04:06.524 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:06.737 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:04:06.737 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:04:07.738 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:04:07.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:07.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:04:07.957 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:04:07.957 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:07.957 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:08.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:04:08.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:04:08.243 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:04:08.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:04:08.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:04:08.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:04:08.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:04:08.244 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:08.459 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:04:08.459 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:04:08.460 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:08.460 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:04:08.742 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:04:08.743 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: stamp=2026-02-25T22:04:04.786843+0000 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: test -n 2026-02-25T22:04:04.786843+0000 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1534: test_get_last_scrub_stamp: teardown td/ceph-helpers 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:08.956 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:08.957 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:08.957 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:08.957 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:08.957 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:09.064 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:09.065 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:09.065 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:09.065 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:09.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:09.066 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:09.067 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:09.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:09.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:09.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:09.068 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:09.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:09.070 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:09.071 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:09.076 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:09.076 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.076 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.076 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_not_primary td/ceph-helpers 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: test_get_not_primary: local dir=td/ceph-helpers 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: test_get_not_primary: setup td/ceph-helpers 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:09.079 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:09.080 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:09.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:09.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:09.080 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:09.080 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:09.080 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:09.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:09.081 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:09.082 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:09.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:09.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:09.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:09.083 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:09.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:09.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:09.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:09.085 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:09.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:09.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:09.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:09.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:09.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:09.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:09.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:09.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:09.092 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:09.092 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.092 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:09.093 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:09.094 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:09.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-25T22:04:09.116 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:09.117 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-25T22:04:09.145 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:09.146 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:09.147 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:09.147 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:09.147 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.148 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.149 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:09.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:09.149 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:09.205 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:09.205 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:09.205 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:09.205 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:09.206 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.207 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:09.208 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:09.208 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:09.262 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: test_get_not_primary: run_mgr td/ceph-helpers x 2026-02-25T22:04:09.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:09.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:09.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:09.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:09.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:09.263 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:09.376 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:09.376 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:09.376 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:09.377 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:09.377 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.377 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:09.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:09.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:09.400 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: test_get_not_primary: run_osd td/ceph-helpers 0 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:09.401 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:09.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:09.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:09.404 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:09.404 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0a275a48-d416-4b37-84a3-67e413a4b465 2026-02-25T22:04:09.404 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0a275a48-d416-4b37-84a3-67e413a4b465' 2026-02-25T22:04:09.404 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 0a275a48-d416-4b37-84a3-67e413a4b465 2026-02-25T22:04:09.405 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:09.415 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZcZ9pYKarGBAAcnuUdp460SOsXnYQu8gVKw== 2026-02-25T22:04:09.415 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZcZ9pYKarGBAAcnuUdp460SOsXnYQu8gVKw=="}' 2026-02-25T22:04:09.415 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0a275a48-d416-4b37-84a3-67e413a4b465 -i td/ceph-helpers/0/new.json 2026-02-25T22:04:09.525 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:09.533 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:09.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZcZ9pYKarGBAAcnuUdp460SOsXnYQu8gVKw== --osd-uuid 0a275a48-d416-4b37-84a3-67e413a4b465 2026-02-25T22:04:09.551 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:09.546+0000 7fc549ab1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:09.552 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:09.546+0000 7fc549ab1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:09.553 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:09.546+0000 7fc549ab1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:09.554 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:09.546+0000 7fc549ab1a40 -1 bdev(0x558fcae52400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:09.554 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:09.550+0000 7fc549ab1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:10.309 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:10.310 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:10.311 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:10.311 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:10.311 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:10.428 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:10.429 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:10.429 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:10.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:10.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:10.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:10.453 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:10.446+0000 7fcf12404a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:10.455 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:10.450+0000 7fcf12404a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:10.456 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:10.450+0000 7fcf12404a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:10.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:10.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:10.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:10.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:10.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:10.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:10.543 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:10.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:10.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:10.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:10.653 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:10.952 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:10.946+0000 7fcf12404a40 -1 Falling back to public interface 2026-02-25T22:04:11.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:11.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:11.655 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:11.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:11.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:11.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:11.866 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:11.882 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:11.878+0000 7fcf12404a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:04:12.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:12.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:12.868 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:12.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:12.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:12.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:13.084 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1426143931,v1:127.0.0.1:6801/1426143931] [v2:127.0.0.1:6802/1426143931,v1:127.0.0.1:6803/1426143931] exists,up 0a275a48-d416-4b37-84a3-67e413a4b465 2026-02-25T22:04:13.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: test_get_not_primary: run_osd td/ceph-helpers 1 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:13.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:13.089 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:13.090 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-25T22:04:13.091 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:13.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=efcea6e9-f4a3-497e-9015-ed9f76bf2732 2026-02-25T22:04:13.092 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 efcea6e9-f4a3-497e-9015-ed9f76bf2732' 2026-02-25T22:04:13.093 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 efcea6e9-f4a3-497e-9015-ed9f76bf2732 2026-02-25T22:04:13.093 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:13.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdcZ9peLj1BRAAj+5QWD/2HoQUoBSE4omWQg== 2026-02-25T22:04:13.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdcZ9peLj1BRAAj+5QWD/2HoQUoBSE4omWQg=="}' 2026-02-25T22:04:13.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new efcea6e9-f4a3-497e-9015-ed9f76bf2732 -i td/ceph-helpers/1/new.json 2026-02-25T22:04:13.328 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:13.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-25T22:04:13.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdcZ9peLj1BRAAj+5QWD/2HoQUoBSE4omWQg== --osd-uuid efcea6e9-f4a3-497e-9015-ed9f76bf2732 2026-02-25T22:04:13.355 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:13.350+0000 7f228e955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:13.357 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:13.350+0000 7f228e955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:13.358 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:13.350+0000 7f228e955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:13.358 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:13.354+0000 7f228e955a40 -1 bdev(0x561e1d60e400 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:13.358 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:13.354+0000 7f228e955a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-25T22:04:13.996 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-25T22:04:13.996 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:13.997 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:04:13.997 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T22:04:13.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:14.295 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T22:04:14.296 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:04:14.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:14.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:14.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:14.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:14.318 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:14.314+0000 7f7ceacd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:14.320 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:14.314+0000 7f7ceacd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:14.321 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:14.314+0000 7f7ceacd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:14.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:04:14.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:14.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:04:14.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:14.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:14.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:14.506 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:14.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:14.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:14.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:04:14.704 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:14.698+0000 7f7ceacd5a40 -1 Falling back to public interface 2026-02-25T22:04:14.719 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:15.398 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:15.390+0000 7f7ceacd5a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:04:15.720 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:15.721 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:15.721 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:15.721 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:15.722 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:15.722 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:04:15.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:16.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:16.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:16.940 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:16.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:16.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:16.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:04:17.145 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/509383545,v1:127.0.0.1:6811/509383545] [v2:127.0.0.1:6812/509383545,v1:127.0.0.1:6813/509383545] exists,up efcea6e9-f4a3-497e-9015-ed9f76bf2732 2026-02-25T22:04:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: test_get_not_primary: create_rbd_pool 2026-02-25T22:04:17.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:04:17.344 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:04:17.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:04:17.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:04:17.591 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:04:17.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:04:18.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:04:19.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: test_get_not_primary: wait_for_clean 2026-02-25T22:04:19.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:04:19.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:04:19.124 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:04:19.124 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:04:19.125 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:04:19.125 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:04:19.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:04:19.125 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:04:19.126 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:04:19.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:04:19.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:04:19.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:04:19.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:04:19.230 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:04:19.230 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:04:19.451 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:04:19.452 INFO:tasks.workunit.client.0.trial163.stderr:1' 2026-02-25T22:04:19.452 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:04:19.452 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:04:19.452 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:04:19.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:04:19.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:04:19.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:04:19.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:04:19.530 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:04:19.608 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-25T22:04:19.608 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-25T22:04:19.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-38654705667' 2026-02-25T22:04:19.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:04:19.609 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:04:19.609 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:04:19.610 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:04:19.611 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:04:19.611 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:04:19.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:04:19.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:04:19.613 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:04:19.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:19.834 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-25T22:04:19.834 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:04:19.835 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705667 2026-02-25T22:04:19.835 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:04:19.836 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:04:19.836 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-25T22:04:19.837 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:04:19.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705667 2026-02-25T22:04:19.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-25T22:04:19.838 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 38654705667 2026-02-25T22:04:19.839 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:04:20.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705666 -lt 38654705667 2026-02-25T22:04:20.057 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:04:21.058 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:04:21.059 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:04:21.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705666 -lt 38654705667 2026-02-25T22:04:21.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:04:22.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:04:22.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:04:22.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705668 -lt 38654705667 2026-02-25T22:04:22.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:04:22.487 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:22.487 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:22.761 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:04:22.761 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:04:22.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:04:22.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:04:22.763 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:04:22.763 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:04:22.763 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:04:22.763 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:22.967 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:04:22.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:04:22.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:22.968 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:23.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:04:23.264 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:04:23.265 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:04:23.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: get_primary rbd GROUP 2026-02-25T22:04:23.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-25T22:04:23.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-25T22:04:23.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:04:23.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:04:23.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local primary=1 2026-02-25T22:04:23.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: get_not_primary rbd GROUP 2026-02-25T22:04:23.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-25T22:04:23.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=GROUP 2026-02-25T22:04:23.483 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd GROUP 2026-02-25T22:04:23.483 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-25T22:04:23.484 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-25T22:04:23.484 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:04:23.484 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:04:23.691 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-25T22:04:23.691 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:04:23.691 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-25T22:04:23.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: local not_primary=0 2026-02-25T22:04:23.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: test 0 '!=' 1 2026-02-25T22:04:23.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: test 0 = 0 -o 0 = 1 2026-02-25T22:04:23.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: teardown td/ceph-helpers 2026-02-25T22:04:23.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:23.908 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:23.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:23.909 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:23.909 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:23.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:23.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:23.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:24.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:24.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:24.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:24.018 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:24.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:24.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:24.020 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:24.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:24.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:24.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:24.022 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:24.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:24.023 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:24.023 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:24.030 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:24.030 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.030 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:24.032 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:24.032 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_num_active_clean td/ceph-helpers 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: test_get_num_active_clean: local dir=td/ceph-helpers 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: test_get_num_active_clean: setup td/ceph-helpers 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:24.033 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:24.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:24.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:24.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:24.034 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:24.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:24.035 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:24.037 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:24.037 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:24.037 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:24.038 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:24.038 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:24.038 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:24.038 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:24.039 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:24.040 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:24.040 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:24.042 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:24.042 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.042 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.042 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:24.043 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:24.043 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:24.043 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:24.044 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:24.044 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.044 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.044 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:24.046 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:24.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:24.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:24.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:24.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:24.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:24.047 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:24.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:24.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:24.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:24.048 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:24.069 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:24.069 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:24.069 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:24.070 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:24.070 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.071 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.071 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:24.071 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:24.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:24.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:24.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:24.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:24.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:24.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:24.101 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:24.102 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:24.156 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:24.157 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:24.157 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:24.157 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:24.157 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:24.158 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:24.158 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:24.158 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:24.158 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:24.158 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.158 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.159 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:24.159 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:24.159 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:24.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: test_get_num_active_clean: run_mgr td/ceph-helpers x 2026-02-25T22:04:24.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:24.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:24.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:24.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:24.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:24.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:24.329 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:24.329 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:24.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: test_get_num_active_clean: run_osd td/ceph-helpers 0 2026-02-25T22:04:24.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:24.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:24.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:24.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:24.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:24.352 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:24.352 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=06a2c537-4684-4d81-a333-f96634900e29 2026-02-25T22:04:24.352 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 06a2c537-4684-4d81-a333-f96634900e29' 2026-02-25T22:04:24.352 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 06a2c537-4684-4d81-a333-f96634900e29 2026-02-25T22:04:24.353 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:24.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDocZ9pTFqOFRAAnzaY/Dw2NJUtExNWLycwaA== 2026-02-25T22:04:24.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDocZ9pTFqOFRAAnzaY/Dw2NJUtExNWLycwaA=="}' 2026-02-25T22:04:24.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 06a2c537-4684-4d81-a333-f96634900e29 -i td/ceph-helpers/0/new.json 2026-02-25T22:04:24.477 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:24.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:24.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDocZ9pTFqOFRAAnzaY/Dw2NJUtExNWLycwaA== --osd-uuid 06a2c537-4684-4d81-a333-f96634900e29 2026-02-25T22:04:24.503 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:24.502+0000 7f49d3063a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:24.504 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:24.502+0000 7f49d3063a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:24.505 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:24.502+0000 7f49d3063a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:24.506 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:24.502+0000 7f49d3063a40 -1 bdev(0x555d08eb2400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:24.506 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:24.502+0000 7f49d3063a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:25.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:25.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:25.282 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:25.283 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:25.283 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:25.402 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:25.402 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:25.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:25.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:25.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:25.403 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:25.427 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:25.426+0000 7f7988158a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:25.429 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:25.426+0000 7f7988158a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:25.431 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:25.430+0000 7f7988158a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:25.509 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:25.510 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:25.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:25.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:25.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:25.899 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:25.898+0000 7f7988158a40 -1 Falling back to public interface 2026-02-25T22:04:26.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:26.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:26.616 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:26.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:26.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:26.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:26.722 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:26.872 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:26.870+0000 7f7988158a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:04:27.724 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:27.724 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:27.724 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:27.725 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:27.725 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:27.725 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:27.935 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/876392519,v1:127.0.0.1:6801/876392519] [v2:127.0.0.1:6802/876392519,v1:127.0.0.1:6803/876392519] exists,up 06a2c537-4684-4d81-a333-f96634900e29 2026-02-25T22:04:27.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:27.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:27.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:27.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1369: test_get_num_active_clean: create_rbd_pool 2026-02-25T22:04:27.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:04:28.136 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:04:28.145 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:04:28.145 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:04:28.381 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:04:28.391 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:04:29.392 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:04:29.709 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1370: test_get_num_active_clean: wait_for_clean 2026-02-25T22:04:29.709 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:04:29.709 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:04:29.710 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:04:29.710 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:04:29.711 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:04:29.711 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:04:29.711 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:04:29.711 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:04:29.711 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:04:29.800 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:04:29.800 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:04:29.800 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:04:29.800 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:04:29.800 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:04:29.800 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:04:30.027 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:04:30.028 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:04:30.028 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:04:30.028 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:04:30.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:04:30.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:04:30.110 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:04:30.110 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:04:30.110 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:30.110 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:04:30.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:04:30.112 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:30.112 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:04:30.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:04:30.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:04:30.114 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:04:30.114 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:30.330 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:04:30.330 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:04:31.331 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:04:31.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:31.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:04:31.546 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:04:31.547 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:31.547 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:04:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:04:31.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:04:31.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:04:31.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:04:31.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:04:31.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:04:31.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:32.028 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:04:32.028 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:04:32.029 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:32.029 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:32.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:04:32.302 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:04:32.302 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:04:32.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: get_num_active_clean 2026-02-25T22:04:32.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:04:32.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:04:32.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:04:32.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:04:32.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:32.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: local num_active_clean=4 2026-02-25T22:04:32.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: test 4 = 4 2026-02-25T22:04:32.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1373: test_get_num_active_clean: teardown td/ceph-helpers 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:32.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:32.613 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:32.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:32.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:32.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:32.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:32.615 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:32.615 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:32.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:32.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:32.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:32.617 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:32.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:32.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:32.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:32.625 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:32.625 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.625 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.625 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:32.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_num_active_or_peered td/ceph-helpers 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1393: test_get_num_active_or_peered: local dir=td/ceph-helpers 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: test_get_num_active_or_peered: setup td/ceph-helpers 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:32.628 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:32.629 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:32.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:32.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:32.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:32.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:32.630 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:32.630 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:32.631 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:32.632 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:32.632 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:32.633 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:32.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:32.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:32.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:32.634 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:32.635 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:32.636 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:32.636 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:32.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:32.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.638 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:32.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:32.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:32.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:32.640 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:32.640 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.640 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.640 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:32.641 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:32.643 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:32.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:32.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:32.663 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:32.664 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:32.664 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.664 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.664 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:32.665 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:32.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:32.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:32.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:32.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:32.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:32.695 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.696 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:32.697 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:32.697 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:32.750 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:32.750 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:32.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:32.751 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:32.752 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:32.752 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:32.752 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:32.752 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:32.752 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:32.753 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.753 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.753 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:32.753 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:32.753 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:32.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2026-02-25T22:04:32.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:32.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:32.807 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:32.807 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:32.807 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:32.807 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:32.919 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:32.919 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:32.919 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:32.919 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:32.919 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.919 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.920 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:32.920 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:32.920 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:32.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:32.943 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:32.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:32.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:32.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:32.945 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:32.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=643bf50d-feed-46e7-b709-512466aff06f 2026-02-25T22:04:32.946 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 643bf50d-feed-46e7-b709-512466aff06f 2026-02-25T22:04:32.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 643bf50d-feed-46e7-b709-512466aff06f' 2026-02-25T22:04:32.946 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:32.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDwcZ9phA7ZOBAAdi2FXLXjPyNHZGoi5gRodQ== 2026-02-25T22:04:32.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDwcZ9phA7ZOBAAdi2FXLXjPyNHZGoi5gRodQ=="}' 2026-02-25T22:04:32.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 643bf50d-feed-46e7-b709-512466aff06f -i td/ceph-helpers/0/new.json 2026-02-25T22:04:33.066 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:33.074 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:33.075 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwcZ9phA7ZOBAAdi2FXLXjPyNHZGoi5gRodQ== --osd-uuid 643bf50d-feed-46e7-b709-512466aff06f 2026-02-25T22:04:33.091 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.090+0000 7f8d5d2cca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:33.093 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.090+0000 7f8d5d2cca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:33.093 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.090+0000 7f8d5d2cca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:33.094 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.090+0000 7f8d5d2cca40 -1 bdev(0x55d9c74ee400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:33.094 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.090+0000 7f8d5d2cca40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:33.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:33.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:33.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:33.823 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:33.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:33.946 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:33.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:33.946 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:33.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:33.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:33.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:33.971 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.970+0000 7fe856b7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:33.973 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.970+0000 7fe856b7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:33.974 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:33.974+0000 7fe856b7aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:34.055 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:34.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:34.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:34.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:34.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:34.375 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:34.374+0000 7fe856b7aa40 -1 Falling back to public interface 2026-02-25T22:04:35.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:35.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:35.164 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:35.164 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:35.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:35.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:35.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:35.328 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:35.326+0000 7fe856b7aa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:04:36.279 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:36.279 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:36.280 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:36.280 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:36.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:36.281 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:36.488 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/68845972,v1:127.0.0.1:6801/68845972] [v2:127.0.0.1:6802/68845972,v1:127.0.0.1:6803/68845972] exists,up 643bf50d-feed-46e7-b709-512466aff06f 2026-02-25T22:04:36.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:36.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:36.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:36.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1399: test_get_num_active_or_peered: create_rbd_pool 2026-02-25T22:04:36.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:04:36.685 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:04:36.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:04:36.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:04:36.928 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:04:36.938 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:04:37.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:04:38.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1400: test_get_num_active_or_peered: wait_for_clean 2026-02-25T22:04:38.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:04:38.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:04:38.219 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:04:38.219 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:04:38.219 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:04:38.220 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:04:38.220 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:04:38.220 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:04:38.220 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:04:38.312 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:04:38.312 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:04:38.312 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:04:38.312 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:04:38.312 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:04:38.312 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:04:38.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:04:38.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:04:38.535 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:04:38.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:04:38.612 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:04:38.613 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:04:38.613 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:04:38.613 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:04:38.613 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:38.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:04:38.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:04:38.615 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:38.615 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:04:38.616 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:04:38.616 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:04:38.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:04:38.617 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:38.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:04:38.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:04:39.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:04:39.824 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:40.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:04:40.036 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:04:40.037 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:40.037 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:40.314 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:04:40.314 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:04:40.314 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:04:40.314 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:04:40.314 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:04:40.315 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:04:40.315 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:04:40.315 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:40.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:04:40.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:04:40.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:40.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: get_num_active_or_peered 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-25T22:04:40.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:41.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: local num_peered=4 2026-02-25T22:04:41.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: test 4 = 4 2026-02-25T22:04:41.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: test_get_num_active_or_peered: teardown td/ceph-helpers 2026-02-25T22:04:41.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:41.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:41.031 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:41.031 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:41.031 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:41.031 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:41.031 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:41.031 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:41.138 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:41.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:41.140 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:41.140 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:41.141 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:41.142 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:41.142 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:41.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:41.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:41.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:41.144 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:41.145 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:41.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:41.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:41.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:41.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.152 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_num_pgs td/ceph-helpers 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1420: test_get_num_pgs: local dir=td/ceph-helpers 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1422: test_get_num_pgs: setup td/ceph-helpers 2026-02-25T22:04:41.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:41.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:41.156 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:41.156 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:41.157 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:41.158 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:41.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:41.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:41.159 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:41.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:41.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:41.161 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:41.161 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:41.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:41.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:41.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:41.164 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:41.165 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.165 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.165 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:41.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:41.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:41.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:41.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:41.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:41.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:41.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:41.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:41.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:41.193 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:41.193 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:41.194 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:41.194 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:41.195 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.195 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.195 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:41.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:41.221 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:41.222 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:41.222 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:41.222 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:41.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:41.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:41.224 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:41.278 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:41.278 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:41.278 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:41.279 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:41.279 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:41.280 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:41.281 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:41.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1424: test_get_num_pgs: run_mgr td/ceph-helpers x 2026-02-25T22:04:41.334 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:41.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:41.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:41.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:41.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:41.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:41.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:41.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:41.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:41.447 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:41.447 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.447 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.447 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:41.448 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:41.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:41.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: test_get_num_pgs: run_osd td/ceph-helpers 0 2026-02-25T22:04:41.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:41.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:41.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:41.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:41.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:41.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:41.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ae56265f-e49f-4b24-9869-f0b295839783 2026-02-25T22:04:41.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ae56265f-e49f-4b24-9869-f0b295839783' 2026-02-25T22:04:41.472 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 ae56265f-e49f-4b24-9869-f0b295839783 2026-02-25T22:04:41.473 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:41.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD5cZ9pP6y4HBAAEdP7TCuR3gqgquoVMe5g3A== 2026-02-25T22:04:41.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD5cZ9pP6y4HBAAEdP7TCuR3gqgquoVMe5g3A=="}' 2026-02-25T22:04:41.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ae56265f-e49f-4b24-9869-f0b295839783 -i td/ceph-helpers/0/new.json 2026-02-25T22:04:41.594 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:41.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:41.603 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5cZ9pP6y4HBAAEdP7TCuR3gqgquoVMe5g3A== --osd-uuid ae56265f-e49f-4b24-9869-f0b295839783 2026-02-25T22:04:41.619 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:41.617+0000 7f0df9828a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:41.620 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:41.617+0000 7f0df9828a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:41.621 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:41.617+0000 7f0df9828a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:41.622 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:41.617+0000 7f0df9828a40 -1 bdev(0x5612db290400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:41.622 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:41.621+0000 7f0df9828a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:42.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:42.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:42.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:42.362 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:42.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:42.482 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:42.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:42.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:42.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:42.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:42.484 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:42.506 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:42.501+0000 7fc176c79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:42.508 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:42.505+0000 7fc176c79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:42.509 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:42.505+0000 7fc176c79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:42.591 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:42.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:42.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:42.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:42.696 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:42.999 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:42.997+0000 7fc176c79a40 -1 Falling back to public interface 2026-02-25T22:04:43.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:43.697 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:43.698 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:43.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:43.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:43.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:43.798 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:43.797+0000 7fc176c79a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:04:43.804 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:44.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:44.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:44.806 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:44.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:44.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:44.806 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:45.016 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1693731415,v1:127.0.0.1:6801/1693731415] [v2:127.0.0.1:6802/1693731415,v1:127.0.0.1:6803/1693731415] exists,up ae56265f-e49f-4b24-9869-f0b295839783 2026-02-25T22:04:45.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:45.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:45.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:45.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1426: test_get_num_pgs: create_rbd_pool 2026-02-25T22:04:45.017 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:04:45.214 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:04:45.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:04:45.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:04:45.458 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:04:45.468 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:04:46.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:04:46.747 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1427: test_get_num_pgs: wait_for_clean 2026-02-25T22:04:46.748 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:04:46.748 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:04:46.748 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:04:46.748 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:04:46.748 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:04:46.748 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:04:46.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:04:46.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:04:46.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:04:46.856 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:04:46.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:04:46.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:04:46.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:04:46.857 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:04:46.857 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:04:47.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:04:47.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:04:47.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:04:47.077 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:04:47.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:04:47.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:04:47.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:04:47.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:04:47.159 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:47.160 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:04:47.160 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:04:47.161 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:04:47.161 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:04:47.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:04:47.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:04:47.163 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:04:47.163 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:47.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:04:47.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:04:48.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:04:48.382 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:04:48.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:04:48.601 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:04:48.602 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:48.602 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:48.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:04:48.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:04:48.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:04:48.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:04:48.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:04:48.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:04:48.891 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:04:48.891 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:04:49.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:04:49.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:04:49.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:49.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:49.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:04:49.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:04:49.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:04:49.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: get_num_pgs 2026-02-25T22:04:49.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:04:49.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: local num_pgs=4 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: test 4 -gt 0 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1430: test_get_num_pgs: teardown td/ceph-helpers 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:49.658 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:49.659 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:49.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:49.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:49.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:49.766 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:49.767 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:49.767 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:49.768 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:49.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:49.769 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:49.769 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:49.770 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:49.770 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:49.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:49.771 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:49.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:49.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:49.772 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:49.776 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:49.776 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:49.776 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:49.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:49.778 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_op_scheduler td/ceph-helpers 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: test_get_op_scheduler: local dir=td/ceph-helpers 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: test_get_op_scheduler: setup td/ceph-helpers 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:49.779 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:49.780 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:49.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:49.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:49.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:49.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:49.781 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:49.782 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:49.782 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:49.783 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:49.783 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:49.784 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:49.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:49.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:49.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:49.786 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:49.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:49.788 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:49.788 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:49.789 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:49.789 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:49.789 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:49.789 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:49.790 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:49.791 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:49.791 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:49.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:49.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:49.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:49.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:49.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: test_get_op_scheduler: run_mon td/ceph-helpers a 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:49.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:49.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:04:49.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:49.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:49.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:49.818 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:49.818 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:49.818 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:49.818 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:49.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:04:49.845 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:49.846 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:49.846 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:49.847 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:49.847 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:49.847 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:49.848 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:49.849 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:49.904 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:49.905 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:49.905 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:49.905 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:49.906 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:49.907 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:49.907 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:49.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: test_get_op_scheduler: run_mgr td/ceph-helpers x 2026-02-25T22:04:49.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:49.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:49.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:49.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:49.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:49.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:50.075 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:50.075 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:50.075 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:50.075 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:50.076 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:50.076 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:50.076 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:50.076 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:50.077 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:50.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2026-02-25T22:04:50.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:50.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:50.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:50.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:50.098 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:50.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=wpq 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:50.100 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:50.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1edb622e-7b84-442f-8066-feb7cea84f5b 2026-02-25T22:04:50.101 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 1edb622e-7b84-442f-8066-feb7cea84f5b 2026-02-25T22:04:50.101 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1edb622e-7b84-442f-8066-feb7cea84f5b' 2026-02-25T22:04:50.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:50.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQACcp9pOqebBhAAmE/ZIhz6qYogi1EDW3JOGQ== 2026-02-25T22:04:50.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQACcp9pOqebBhAAmE/ZIhz6qYogi1EDW3JOGQ=="}' 2026-02-25T22:04:50.112 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1edb622e-7b84-442f-8066-feb7cea84f5b -i td/ceph-helpers/0/new.json 2026-02-25T22:04:50.224 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:50.233 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:50.234 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQACcp9pOqebBhAAmE/ZIhz6qYogi1EDW3JOGQ== --osd-uuid 1edb622e-7b84-442f-8066-feb7cea84f5b 2026-02-25T22:04:50.251 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:50.249+0000 7f4ff9a5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:50.252 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:50.249+0000 7f4ff9a5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:50.254 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:50.249+0000 7f4ff9a5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:50.254 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:50.253+0000 7f4ff9a5ba40 -1 bdev(0x5646bb3d6400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:50.254 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:50.253+0000 7f4ff9a5ba40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:51.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:51.153 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:51.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:51.154 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:51.155 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:51.274 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:51.275 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:51.275 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2026-02-25T22:04:51.275 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:51.275 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:51.276 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:51.298 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:51.293+0000 7ff6b6f26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:51.300 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:51.297+0000 7ff6b6f26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:51.301 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:51.297+0000 7ff6b6f26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:51.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:51.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:51.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:51.382 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:51.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:51.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:51.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:51.383 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:51.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:51.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:51.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:51.823 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:51.821+0000 7ff6b6f26a40 -1 Falling back to public interface 2026-02-25T22:04:52.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:52.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:52.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:52.488 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:52.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:52.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:52.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:52.704 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:52.701+0000 7ff6b6f26a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:04:53.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:53.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:53.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:04:53.693 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:04:53.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:53.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:53.904 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2346428830,v1:127.0.0.1:6801/2346428830] [v2:127.0.0.1:6802/2346428830,v1:127.0.0.1:6803/2346428830] exists,up 1edb622e-7b84-442f-8066-feb7cea84f5b 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_get_op_scheduler: get_op_scheduler 0 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: get_op_scheduler: local id=0 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: get_op_scheduler: get_config osd 0 osd_op_queue 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-25T22:04:53.906 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-25T22:04:53.907 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.0.asok 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:53.908 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-osd.0.asok config get osd_op_queue 2026-02-25T22:04:53.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_get_op_scheduler: test wpq = wpq 2026-02-25T22:04:53.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-25T22:04:53.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:53.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-25T22:04:53.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:53.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=78b123aa-fce3-4528-82a1-7fa02eb21806 2026-02-25T22:04:53.975 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 78b123aa-fce3-4528-82a1-7fa02eb21806 2026-02-25T22:04:53.976 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 78b123aa-fce3-4528-82a1-7fa02eb21806' 2026-02-25T22:04:53.976 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:53.985 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFcp9p4hunOhAA9YkBo7BON0Yd1Tqr2KIOpQ== 2026-02-25T22:04:53.985 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFcp9p4hunOhAA9YkBo7BON0Yd1Tqr2KIOpQ=="}' 2026-02-25T22:04:53.985 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 78b123aa-fce3-4528-82a1-7fa02eb21806 -i td/ceph-helpers/1/new.json 2026-02-25T22:04:54.204 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:54.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-25T22:04:54.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAFcp9p4hunOhAA9YkBo7BON0Yd1Tqr2KIOpQ== --osd-uuid 78b123aa-fce3-4528-82a1-7fa02eb21806 2026-02-25T22:04:54.231 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:54.229+0000 7feff5793a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:54.232 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:54.229+0000 7feff5793a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:54.233 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:54.229+0000 7feff5793a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:54.234 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:54.233+0000 7feff5793a40 -1 bdev(0x55fe1616c400 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:54.234 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:54.233+0000 7feff5793a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-25T22:04:54.888 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-25T22:04:54.889 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:54.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:04:54.890 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T22:04:54.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:55.170 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T22:04:55.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:04:55.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T22:04:55.171 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:55.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:55.172 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:55.193 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:55.189+0000 7ff701083a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:55.195 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:55.193+0000 7ff701083a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:55.196 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:55.193+0000 7ff701083a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:55.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:04:55.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:55.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:04:55.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:55.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:55.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:55.384 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:55.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:55.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:55.385 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:04:55.516 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:55.513+0000 7ff701083a40 -1 Falling back to public interface 2026-02-25T22:04:55.589 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:56.109 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:56.105+0000 7ff701083a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:04:56.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:56.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:56.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:56.592 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:56.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:56.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:04:56.812 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2431567438,v1:127.0.0.1:6811/2431567438] [v2:127.0.0.1:6812/2431567438,v1:127.0.0.1:6813/2431567438] exists,up 78b123aa-fce3-4528-82a1-7fa02eb21806 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: test_get_op_scheduler: get_op_scheduler 1 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: get_op_scheduler: local id=1 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: get_op_scheduler: get_config osd 1 osd_op_queue 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=1 2026-02-25T22:04:56.813 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-25T22:04:56.814 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.1 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-osd.1.asok 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:56.815 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-osd.1.asok config get osd_op_queue 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: test_get_op_scheduler: teardown td/ceph-helpers 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:56.890 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:56.891 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:56.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:56.999 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:57.000 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:57.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:57.002 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:57.002 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:57.003 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:57.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:57.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:57.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:57.005 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:57.005 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:57.006 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:57.006 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:57.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:57.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.011 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:57.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:57.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:57.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:04:57.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2026-02-25T22:04:57.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1450: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-25T22:04:57.013 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1452: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:04:57.014 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:04:57.015 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:04:57.015 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:04:57.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:04:57.016 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:04:57.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:04:57.018 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:04:57.018 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:04:57.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:57.019 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:04:57.020 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:04:57.020 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:04:57.021 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:04:57.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:04:57.022 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:04:57.023 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:04:57.023 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.024 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:04:57.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:04:57.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:04:57.025 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:04:57.026 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:04:57.026 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.026 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.026 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:04:57.028 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1453: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:04:57.029 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:04:57.030 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:57.050 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:04:57.050 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:57.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:57.051 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:57.051 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.051 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.051 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:57.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:04:57.081 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:04:57.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:04:57.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:57.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:57.082 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:04:57.083 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:04:57.083 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:57.084 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:57.085 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:04:57.138 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:04:57.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:04:57.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:04:57.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.140 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.141 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:04:57.141 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:04:57.141 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:04:57.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1454: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-25T22:04:57.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:04:57.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:04:57.196 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:04:57.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:04:57.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:04:57.197 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:04:57.311 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:04:57.311 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:57.312 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:57.312 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:57.312 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.312 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.313 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:57.313 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:57.314 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:04:57.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:04:57.336 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:04:57.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:04:57.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:04:57.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:04:57.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:04:57.338 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:04:57.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=817dcffb-95b8-46d1-b640-912e29e62730 2026-02-25T22:04:57.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 817dcffb-95b8-46d1-b640-912e29e62730' 2026-02-25T22:04:57.339 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 817dcffb-95b8-46d1-b640-912e29e62730 2026-02-25T22:04:57.339 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:04:57.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAJcp9pyZW0FBAAUNjCU0Fbvs0gII8h4654FQ== 2026-02-25T22:04:57.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAJcp9pyZW0FBAAUNjCU0Fbvs0gII8h4654FQ=="}' 2026-02-25T22:04:57.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 817dcffb-95b8-46d1-b640-912e29e62730 -i td/ceph-helpers/0/new.json 2026-02-25T22:04:57.463 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:57.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:04:57.472 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJcp9pyZW0FBAAUNjCU0Fbvs0gII8h4654FQ== --osd-uuid 817dcffb-95b8-46d1-b640-912e29e62730 2026-02-25T22:04:57.488 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:57.485+0000 7fd1d8ccea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:57.490 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:57.485+0000 7fd1d8ccea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:57.491 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:57.489+0000 7fd1d8ccea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:57.491 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:57.489+0000 7fd1d8ccea40 -1 bdev(0x55a6ab5ac400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:04:57.491 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:57.489+0000 7fd1d8ccea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:04:58.242 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:04:58.242 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:04:58.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:04:58.243 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:04:58.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:04:58.361 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:04:58.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:04:58.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:04:58.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:04:58.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:04:58.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:04:58.386 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:58.385+0000 7f9763cf2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:58.389 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:58.385+0000 7f9763cf2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:58.390 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:58.389+0000 7f9763cf2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:04:58.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:04:58.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:04:58.469 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:04:58.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:04:58.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:04:58.470 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:58.470 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:04:58.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:04:58.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:58.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:58.584 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:58.883 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:58.881+0000 7f9763cf2a40 -1 Falling back to public interface 2026-02-25T22:04:59.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:04:59.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:04:59.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:04:59.586 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:04:59.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:04:59.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:04:59.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:04:59.855 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:04:59.853+0000 7f9763cf2a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:05:00.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:00.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:00.695 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:05:00.696 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:05:00.696 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:00.696 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:00.908 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/115222211,v1:127.0.0.1:6801/115222211] [v2:127.0.0.1:6802/115222211,v1:127.0.0.1:6803/115222211] exists,up 817dcffb-95b8-46d1-b640-912e29e62730 2026-02-25T22:05:00.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:00.909 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:00.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:00.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1456: test_get_osd_id_used_by_pgs: create_rbd_pool 2026-02-25T22:05:00.910 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:05:01.122 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:05:01.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:05:01.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:05:01.351 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:05:01.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:05:02.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:05:02.641 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1457: test_get_osd_id_used_by_pgs: wait_for_clean 2026-02-25T22:05:02.642 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:05:02.642 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:05:02.642 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:05:02.642 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:05:02.643 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:02.643 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:02.643 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:02.643 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:02.643 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:02.747 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:05:02.747 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:05:02.747 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:05:02.747 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:05:02.747 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:05:02.748 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:05:02.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:05:02.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:05:02.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:05:02.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:05:03.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:05:03.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:05:03.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:05:03.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:05:03.034 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:05:03.034 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:05:03.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:05:03.036 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:05:03.036 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:05:03.037 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:05:03.037 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:05:03.037 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:05:03.038 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:03.256 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:05:03.257 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:05:04.258 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:05:04.258 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:04.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:05:04.478 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:05:04.479 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:04.479 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:04.770 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:05:04.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:05:04.771 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:05:04.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:05:04.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:05:04.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:05:04.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:05:04.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:05:04.984 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:05:04.985 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:05:04.985 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:04.985 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:05.274 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:05:05.274 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:05:05.274 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:05:05.275 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-25T22:05:05.275 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: uniq 2026-02-25T22:05:05.275 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-25T22:05:05.275 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-25T22:05:05.276 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: local osd_ids=0 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: test 0 = 0 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1460: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:05.493 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:05.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:05.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:05.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:05.601 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:05.601 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:05.602 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:05.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:05.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:05.604 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:05.604 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:05.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:05.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:05.606 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:05.606 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:05.607 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:05.608 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:05.609 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:05.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:05.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.614 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.614 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_osds td/ceph-helpers 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: test_get_osds: local dir=td/ceph-helpers 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1024: test_get_osds: setup td/ceph-helpers 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:05:05.617 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:05.618 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:05.619 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:05.619 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:05.620 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:05.620 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:05.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:05.621 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:05.622 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:05.623 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:05.623 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:05.623 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:05.623 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:05.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:05.625 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:05.626 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:05.627 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:05.627 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.627 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:05.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:05.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:05.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:05:05.630 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:05:05.630 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.630 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.630 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:05:05.632 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:05:05.633 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:05:05.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:05:05.634 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-25T22:05:05.654 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:05:05.654 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:05.655 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:05.655 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:05.655 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.655 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.655 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:05.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-25T22:05:05.688 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:05:05.689 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:05:05.690 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:05.690 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:05.690 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.691 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.692 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:05.692 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:05.692 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:05:05.747 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:05:05.747 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:05.747 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:05.748 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.749 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:05.750 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:05.750 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1026: test_get_osds: run_mgr td/ceph-helpers x 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:05:05.805 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:05:05.919 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:05:05.919 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:05.919 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:05.920 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:05.920 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.920 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.920 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:05.921 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:05.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: test_get_osds: run_osd td/ceph-helpers 0 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:05.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:05.943 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:05.944 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:05:05.945 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:05.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=687b7e7c-b754-48cf-b8cb-680d1a955b10 2026-02-25T22:05:05.945 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 687b7e7c-b754-48cf-b8cb-680d1a955b10' 2026-02-25T22:05:05.946 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 687b7e7c-b754-48cf-b8cb-680d1a955b10 2026-02-25T22:05:05.946 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:05.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQARcp9pB3vjOBAA7bfCKGNAHUGPEoNkRXWLCQ== 2026-02-25T22:05:05.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQARcp9pB3vjOBAA7bfCKGNAHUGPEoNkRXWLCQ=="}' 2026-02-25T22:05:05.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 687b7e7c-b754-48cf-b8cb-680d1a955b10 -i td/ceph-helpers/0/new.json 2026-02-25T22:05:06.065 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:06.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:05:06.074 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARcp9pB3vjOBAA7bfCKGNAHUGPEoNkRXWLCQ== --osd-uuid 687b7e7c-b754-48cf-b8cb-680d1a955b10 2026-02-25T22:05:06.091 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.089+0000 7fd9b9d1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:06.092 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.089+0000 7fd9b9d1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:06.094 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.093+0000 7fd9b9d1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:06.094 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.093+0000 7fd9b9d1aa40 -1 bdev(0x5583135da400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:06.094 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.093+0000 7fd9b9d1aa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:05:06.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:05:06.826 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:06.827 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:05:06.827 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:05:06.828 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:06.952 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:05:06.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:05:06.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:06.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:06.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:06.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:06.976 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.973+0000 7f2c2ac10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:06.979 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.977+0000 7f2c2ac10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:06.980 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:06.977+0000 7f2c2ac10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:07.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:05:07.067 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:07.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:05:07.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:07.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:07.068 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:07.068 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:07.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:07.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:07.069 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:07.177 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:07.332 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:07.329+0000 7f2c2ac10a40 -1 Falling back to public interface 2026-02-25T22:05:08.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:08.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:08.179 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:08.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:05:08.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:08.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:08.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:08.379 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:08.377+0000 7f2c2ac10a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:05:09.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:09.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:09.290 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:05:09.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:05:09.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:09.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:09.500 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1714148971,v1:127.0.0.1:6801/1714148971] [v2:127.0.0.1:6802/1714148971,v1:127.0.0.1:6803/1714148971] exists,up 687b7e7c-b754-48cf-b8cb-680d1a955b10 2026-02-25T22:05:09.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:09.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:09.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:09.503 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1028: test_get_osds: run_osd td/ceph-helpers 1 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-25T22:05:09.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:09.505 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:09.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-25T22:05:09.507 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:09.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7f9c164b-94b5-4998-b513-f7f4a995438a 2026-02-25T22:05:09.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7f9c164b-94b5-4998-b513-f7f4a995438a' 2026-02-25T22:05:09.507 INFO:tasks.workunit.client.0.trial163.stdout:add osd1 7f9c164b-94b5-4998-b513-f7f4a995438a 2026-02-25T22:05:09.508 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:09.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAVcp9pjPbAHhAAD9HtZ+bc12dKBDP6U7ywIQ== 2026-02-25T22:05:09.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAVcp9pjPbAHhAAD9HtZ+bc12dKBDP6U7ywIQ=="}' 2026-02-25T22:05:09.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7f9c164b-94b5-4998-b513-f7f4a995438a -i td/ceph-helpers/1/new.json 2026-02-25T22:05:09.733 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:09.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-25T22:05:09.745 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVcp9pjPbAHhAAD9HtZ+bc12dKBDP6U7ywIQ== --osd-uuid 7f9c164b-94b5-4998-b513-f7f4a995438a 2026-02-25T22:05:09.768 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:09.765+0000 7ff80935ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:09.770 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:09.769+0000 7ff80935ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:09.771 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:09.769+0000 7ff80935ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:09.772 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:09.769+0000 7ff80935ba40 -1 bdev(0x5646d3f50400 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:09.772 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:09.769+0000 7ff80935ba40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-25T22:05:10.429 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-25T22:05:10.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:10.430 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:05:10.431 INFO:tasks.workunit.client.0.trial163.stdout:adding osd1 key to auth repository 2026-02-25T22:05:10.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:10.729 INFO:tasks.workunit.client.0.trial163.stdout:start osd.1 2026-02-25T22:05:10.729 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:05:10.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:10.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:10.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:10.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:10.752 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:10.749+0000 7fdf62840a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:10.754 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:10.753+0000 7fdf62840a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:10.755 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:10.753+0000 7fdf62840a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:10.939 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:10.940 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:10.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:10.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:05:11.123 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:11.121+0000 7fdf62840a40 -1 Falling back to public interface 2026-02-25T22:05:11.154 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:11.739 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:11.737+0000 7fdf62840a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:05:12.156 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:12.156 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:12.156 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:05:12.156 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:12.157 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:12.157 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:05:12.375 INFO:tasks.workunit.client.0.trial163.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1087193901,v1:127.0.0.1:6811/1087193901] [v2:127.0.0.1:6812/1087193901,v1:127.0.0.1:6813/1087193901] exists,up 7f9c164b-94b5-4998-b513-f7f4a995438a 2026-02-25T22:05:12.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:12.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:12.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:12.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: test_get_osds: create_rbd_pool 2026-02-25T22:05:12.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:05:12.583 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:05:12.593 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:05:12.593 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:05:12.876 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:05:12.886 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:05:13.887 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:05:14.169 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: test_get_osds: wait_for_clean 2026-02-25T22:05:14.169 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:05:14.169 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:05:14.169 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:05:14.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:05:14.170 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:14.170 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:14.170 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:14.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:14.171 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:14.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:05:14.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:05:14.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:05:14.270 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:05:14.271 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:05:14.271 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:05:14.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:05:14.483 INFO:tasks.workunit.client.0.trial163.stderr:1' 2026-02-25T22:05:14.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:05:14.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:05:14.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:05:14.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T22:05:14.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T22:05:14.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T22:05:14.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:05:14.566 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:05:14.645 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-25T22:05:14.646 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-25T22:05:14.646 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-25T22:05:14.646 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:05:14.646 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T22:05:14.646 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:05:14.647 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:05:14.647 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T22:05:14.647 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:05:14.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T22:05:14.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T22:05:14.649 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836484 2026-02-25T22:05:14.650 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:14.866 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836484 2026-02-25T22:05:14.866 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:05:15.867 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:05:15.868 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:16.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836484 2026-02-25T22:05:16.086 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:05:17.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:05:17.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:17.300 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T22:05:17.300 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:05:17.301 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705667 2026-02-25T22:05:17.301 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:05:17.302 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:05:17.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-25T22:05:17.303 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:05:17.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705667 2026-02-25T22:05:17.305 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.1 seq 38654705667 2026-02-25T22:05:17.305 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-25T22:05:17.305 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:05:17.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-25T22:05:17.520 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:05:17.520 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:17.520 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:17.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:05:17.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:05:17.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:05:17.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:05:17.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:05:17.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:05:17.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:05:17.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:05:18.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:05:18.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:05:18.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:18.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:18.313 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:05:18.313 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:05:18.313 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:05:18.313 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: create_rbd_pool 2026-02-25T22:05:18.314 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:05:18.561 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:05:18.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:05:18.571 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:05:18.841 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:05:18.850 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:05:19.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:05:20.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: get_osds rbd GROUP 2026-02-25T22:05:20.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=rbd 2026-02-25T22:05:20.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=GROUP 2026-02-25T22:05:20.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2026-02-25T22:05:20.134 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map rbd GROUP 2026-02-25T22:05:20.134 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:05:20.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-25T22:05:20.348 INFO:tasks.workunit.client.0.trial163.stderr:0' 2026-02-25T22:05:20.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-25T22:05:20.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: teardown td/ceph-helpers 2026-02-25T22:05:20.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:20.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:20.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:20.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:20.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:20.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:20.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:20.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:20.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:20.458 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:20.459 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:20.459 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:20.460 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:20.460 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:20.460 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:20.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:20.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:20.462 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:20.462 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:20.463 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:20.464 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:20.464 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:20.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:20.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:20.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:20.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:20.473 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_pg td/ceph-helpers 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1091: test_get_pg: local dir=td/ceph-helpers 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: test_get_pg: setup td/ceph-helpers 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:20.474 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:20.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:20.475 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:20.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:20.476 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:20.476 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:20.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:20.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:20.478 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:20.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:20.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:20.479 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:20.479 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:20.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:20.481 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:20.481 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:20.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:20.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.483 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:20.484 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:20.484 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:20.484 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:05:20.485 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:05:20.485 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.485 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:05:20.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:05:20.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:05:20.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:20.509 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:05:20.509 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:20.510 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:20.510 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:20.510 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.510 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.510 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:20.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:20.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:05:20.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:05:20.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:20.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:20.539 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:05:20.540 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:05:20.540 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:20.540 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:20.540 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:20.541 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:20.541 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.541 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.541 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:20.541 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:20.541 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:05:20.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:05:20.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:20.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:20.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:05:20.596 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:05:20.597 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:20.597 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:20.597 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:20.598 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:20.598 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.598 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.598 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:20.598 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:20.598 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:05:20.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1095: test_get_pg: run_mgr td/ceph-helpers x 2026-02-25T22:05:20.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:05:20.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:05:20.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:05:20.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:05:20.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:05:20.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:05:20.761 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:05:20.761 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:20.761 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:20.762 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:20.762 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.762 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.762 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:20.763 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:20.764 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:20.784 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: test_get_pg: run_osd td/ceph-helpers 0 2026-02-25T22:05:20.784 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:05:20.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:20.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:20.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:05:20.788 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:20.788 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04f3bf34-b75b-41f9-9e35-d14aba458060 2026-02-25T22:05:20.788 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 04f3bf34-b75b-41f9-9e35-d14aba458060 2026-02-25T22:05:20.789 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 04f3bf34-b75b-41f9-9e35-d14aba458060' 2026-02-25T22:05:20.789 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:20.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAgcp9pe+qaLxAAgpByCSKKAxVI9Yhus09JgQ== 2026-02-25T22:05:20.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAgcp9pe+qaLxAAgpByCSKKAxVI9Yhus09JgQ=="}' 2026-02-25T22:05:20.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04f3bf34-b75b-41f9-9e35-d14aba458060 -i td/ceph-helpers/0/new.json 2026-02-25T22:05:20.912 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:20.919 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:05:20.921 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgcp9pe+qaLxAAgpByCSKKAxVI9Yhus09JgQ== --osd-uuid 04f3bf34-b75b-41f9-9e35-d14aba458060 2026-02-25T22:05:20.936 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:20.933+0000 7fd45545ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:20.939 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:20.937+0000 7fd45545ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:20.940 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:20.937+0000 7fd45545ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:20.940 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:20.937+0000 7fd45545ca40 -1 bdev(0x55b0c9e9e400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:20.940 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:20.937+0000 7fd45545ca40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:05:21.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:05:21.698 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:21.699 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:05:21.700 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:05:21.700 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:21.819 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:05:21.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:05:21.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:21.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:21.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:21.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:21.842 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:21.841+0000 7fd4c91cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:21.844 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:21.841+0000 7fd4c91cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:21.845 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:21.841+0000 7fd4c91cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:21.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:05:21.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:21.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:05:21.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:21.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:21.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:21.928 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:21.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:21.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:21.928 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:22.034 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:22.379 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:22.377+0000 7fd4c91cda40 -1 Falling back to public interface 2026-02-25T22:05:23.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:23.035 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:23.036 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:23.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:05:23.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:23.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:23.145 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:23.369 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:23.365+0000 7fd4c91cda40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:05:24.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:24.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:24.147 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:05:24.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:05:24.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:24.148 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:24.357 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3032465279,v1:127.0.0.1:6801/3032465279] [v2:127.0.0.1:6802/3032465279,v1:127.0.0.1:6803/3032465279] exists,up 04f3bf34-b75b-41f9-9e35-d14aba458060 2026-02-25T22:05:24.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:24.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:24.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:24.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1097: test_get_pg: create_rbd_pool 2026-02-25T22:05:24.359 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:05:24.555 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:05:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:05:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:05:24.808 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:05:24.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:05:25.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:05:26.146 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: test_get_pg: wait_for_clean 2026-02-25T22:05:26.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:05:26.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:05:26.147 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:05:26.147 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:05:26.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:26.148 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:26.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:26.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:26.148 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:26.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:05:26.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:05:26.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:05:26.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:05:26.239 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:05:26.239 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:05:26.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:05:26.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:05:26.450 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:05:26.451 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:05:26.532 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T22:05:26.533 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T22:05:26.533 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T22:05:26.533 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:05:26.533 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T22:05:26.533 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:05:26.534 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:05:26.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T22:05:26.535 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:05:26.536 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T22:05:26.536 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836484 2026-02-25T22:05:26.537 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T22:05:26.537 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:26.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-25T22:05:26.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:05:27.745 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:05:27.746 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:27.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T22:05:27.965 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:05:27.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:27.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:28.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:05:28.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:05:28.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:05:28.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:05:28.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:05:28.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:05:28.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:05:28.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:05:28.467 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:05:28.468 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:05:28.468 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:28.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:28.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:05:28.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:05:28.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:05:28.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: get_pg rbd GROUP 2026-02-25T22:05:28.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-25T22:05:28.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=GROUP 2026-02-25T22:05:28.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2026-02-25T22:05:28.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd GROUP 2026-02-25T22:05:28.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-25T22:05:28.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: teardown td/ceph-helpers 2026-02-25T22:05:28.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:28.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:28.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:28.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:28.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:28.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:28.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:28.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:29.083 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:29.083 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:29.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:29.084 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:29.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:29.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:29.086 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:29.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:29.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:29.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:29.087 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:29.088 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:29.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:29.089 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:29.095 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:29.095 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.095 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:29.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:29.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:29.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:29.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_primary td/ceph-helpers 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: test_get_primary: local dir=td/ceph-helpers 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: test_get_primary: setup td/ceph-helpers 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:29.098 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:29.099 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:29.099 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:29.100 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:29.101 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:29.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:29.102 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:29.102 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:29.103 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:29.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:29.104 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:29.104 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:29.105 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:29.106 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:29.106 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:29.107 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:29.107 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.108 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:29.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:29.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:29.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:05:29.110 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:05:29.111 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.111 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.111 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:05:29.112 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:05:29.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:05:29.113 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:05:29.114 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:29.135 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:05:29.135 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:29.135 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:29.135 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:29.136 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.136 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.136 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:29.136 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:29.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:05:29.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:05:29.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:29.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:29.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:05:29.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:05:29.168 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:29.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:05:29.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:05:29.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:29.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:29.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:05:29.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:05:29.224 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:29.225 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: test_get_primary: local osd=0 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: test_get_primary: run_mgr td/ceph-helpers x 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:05:29.278 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:05:29.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:05:29.387 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:29.388 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:29.388 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:29.388 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.388 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.388 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:29.388 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:29.389 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: test_get_primary: run_osd td/ceph-helpers 0 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:05:29.410 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:29.411 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:29.412 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:29.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:29.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:29.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:05:29.413 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:29.413 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a26d29d-b8a9-488e-b0fc-08868b6c6e46 2026-02-25T22:05:29.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3a26d29d-b8a9-488e-b0fc-08868b6c6e46' 2026-02-25T22:05:29.414 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 3a26d29d-b8a9-488e-b0fc-08868b6c6e46 2026-02-25T22:05:29.414 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:29.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQApcp9pkqguGRAANnALilkAnxhCXq0EElcWdw== 2026-02-25T22:05:29.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQApcp9pkqguGRAANnALilkAnxhCXq0EElcWdw=="}' 2026-02-25T22:05:29.423 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a26d29d-b8a9-488e-b0fc-08868b6c6e46 -i td/ceph-helpers/0/new.json 2026-02-25T22:05:29.538 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:29.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:05:29.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApcp9pkqguGRAANnALilkAnxhCXq0EElcWdw== --osd-uuid 3a26d29d-b8a9-488e-b0fc-08868b6c6e46 2026-02-25T22:05:29.563 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:29.561+0000 7fd8d793aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:29.565 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:29.561+0000 7fd8d793aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:29.566 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:29.565+0000 7fd8d793aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:29.567 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:29.565+0000 7fd8d793aa40 -1 bdev(0x55cc4902c400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:29.567 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:29.565+0000 7fd8d793aa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:05:30.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:05:30.320 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:30.321 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:05:30.322 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:05:30.322 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:30.440 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:05:30.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:05:30.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:30.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:30.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:30.442 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:30.465 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:30.461+0000 7f93b49eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:30.468 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:30.465+0000 7f93b49eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:30.469 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:30.465+0000 7f93b49eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:30.547 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:30.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:30.548 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:30.653 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:30.995 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:30.993+0000 7f93b49eaa40 -1 Falling back to public interface 2026-02-25T22:05:31.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:31.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:31.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:05:31.655 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:31.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:31.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:31.760 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:31.956 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:31.953+0000 7f93b49eaa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:05:32.761 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:32.762 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:32.762 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:05:32.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:05:32.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:32.763 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:32.973 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/656048029,v1:127.0.0.1:6801/656048029] [v2:127.0.0.1:6802/656048029,v1:127.0.0.1:6803/656048029] exists,up 3a26d29d-b8a9-488e-b0fc-08868b6c6e46 2026-02-25T22:05:32.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:32.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:32.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:32.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: test_get_primary: create_rbd_pool 2026-02-25T22:05:32.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:05:33.169 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:05:33.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:05:33.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:05:33.420 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:05:33.431 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:05:34.433 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:05:34.711 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: test_get_primary: wait_for_clean 2026-02-25T22:05:34.711 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:05:34.711 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:05:34.711 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:05:34.711 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:05:34.712 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:34.712 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:34.712 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:34.713 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:34.713 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:34.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:05:34.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:05:34.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:05:34.820 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:05:34.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:05:34.821 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:05:35.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:05:35.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:05:35.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:05:35.037 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:05:35.115 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:05:35.115 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:05:35.115 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:05:35.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:05:35.116 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:05:35.116 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:05:35.117 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:05:35.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:05:35.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:05:35.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:05:35.119 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:05:35.120 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:05:35.120 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:35.345 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:05:35.345 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:05:36.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:05:36.347 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:36.554 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:05:36.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:05:36.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:36.555 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:36.834 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:05:36.835 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:05:36.835 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:05:36.836 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:05:36.836 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:05:36.836 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:05:36.836 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:05:36.836 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:05:37.037 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:05:37.037 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:05:37.038 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:37.038 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:37.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:05:37.326 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:05:37.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:05:37.327 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: get_primary rbd GROUP 2026-02-25T22:05:37.327 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-25T22:05:37.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-25T22:05:37.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:05:37.328 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:05:37.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: test 0 = 0 2026-02-25T22:05:37.542 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: test_get_primary: teardown td/ceph-helpers 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:37.543 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:37.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:37.651 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:37.652 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:37.652 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:37.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:37.654 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:37.654 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:37.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:37.655 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:37.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:37.656 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:37.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:37.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:37.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:37.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:37.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:37.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:37.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:37.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:37.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:37.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:37.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_timeout_delays td/ceph-helpers 2026-02-25T22:05:37.664 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 2026-02-25T22:05:37.665 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.665 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.678 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '1 ' = '1 ' 2026-02-25T22:05:37.678 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 5 2026-02-25T22:05:37.679 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.679 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.679 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.680 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2026-02-25T22:05:37.704 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 6 2026-02-25T22:05:37.705 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.705 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.706 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.730 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2026-02-25T22:05:37.730 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 7 2026-02-25T22:05:37.731 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.731 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.732 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.732 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.732 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.756 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2026-02-25T22:05:37.757 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 8 2026-02-25T22:05:37.758 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.758 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.758 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.758 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.758 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.790 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2026-02-25T22:05:37.790 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 1 .1 2026-02-25T22:05:37.791 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.791 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2026-02-25T22:05:37.818 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 1.5 .1 2026-02-25T22:05:37.819 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.819 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.820 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.820 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.820 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.853 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2026-02-25T22:05:37.854 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 5 .1 2026-02-25T22:05:37.855 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.855 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.855 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.855 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.855 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.905 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2026-02-25T22:05:37.905 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: get_timeout_delays 6 .1 2026-02-25T22:05:37.906 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.906 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.907 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.907 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.907 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:37.957 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2026-02-25T22:05:37.958 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: get_timeout_delays 6.3 .1 2026-02-25T22:05:37.958 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:37.959 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:37.959 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:37.959 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:37.959 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:38.009 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2026-02-25T22:05:38.009 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: get_timeout_delays 20 .1 2026-02-25T22:05:38.010 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:38.010 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:38.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:38.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:38.011 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:38.078 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2026-02-25T22:05:38.079 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: get_timeout_delays 300 .1 0 2026-02-25T22:05:38.079 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:38.079 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:38.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:38.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:38.080 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:38.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2026-02-25T22:05:38.163 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: get_timeout_delays 300 .1 10 2026-02-25T22:05:38.164 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:38.164 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:38.164 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:38.164 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:38.164 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:38.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2026-02-25T22:05:38.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:38.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_is_clean td/ceph-helpers 2026-02-25T22:05:38.477 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: test_is_clean: local dir=td/ceph-helpers 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: test_is_clean: setup td/ceph-helpers 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:38.478 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:38.479 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:38.479 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:38.480 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:38.480 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:38.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:38.482 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:38.482 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:38.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:38.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:38.484 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:38.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:38.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:38.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:38.486 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:38.487 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:38.488 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.488 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.488 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:38.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:38.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:38.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:05:38.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:05:38.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.491 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.491 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:05:38.492 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:05:38.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:05:38.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:05:38.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:05:38.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:38.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:05:38.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:38.515 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:38.515 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:38.515 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.515 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.516 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:38.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:38.544 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:05:38.545 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:05:38.545 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:38.545 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:38.545 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:05:38.546 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:38.547 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:05:38.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:05:38.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:38.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:38.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:05:38.603 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:05:38.604 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:38.605 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:05:38.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: test_is_clean: run_mgr td/ceph-helpers x 2026-02-25T22:05:38.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:05:38.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:05:38.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:05:38.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:05:38.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:05:38.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:05:38.774 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:05:38.774 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:38.774 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:38.775 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:38.775 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.775 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.775 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:38.776 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:38.776 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:38.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: test_is_clean: run_osd td/ceph-helpers 0 2026-02-25T22:05:38.796 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:05:38.797 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:38.798 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:38.799 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:05:38.800 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:38.800 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36f7b523-422c-4d1f-a7dd-37a53f244759 2026-02-25T22:05:38.800 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 36f7b523-422c-4d1f-a7dd-37a53f244759 2026-02-25T22:05:38.801 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 36f7b523-422c-4d1f-a7dd-37a53f244759' 2026-02-25T22:05:38.801 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:38.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAycp9pAYooMBAAjE0EdEiKI6AMQFcNKtsYMg== 2026-02-25T22:05:38.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAycp9pAYooMBAAjE0EdEiKI6AMQFcNKtsYMg=="}' 2026-02-25T22:05:38.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36f7b523-422c-4d1f-a7dd-37a53f244759 -i td/ceph-helpers/0/new.json 2026-02-25T22:05:38.918 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:38.926 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:05:38.927 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAycp9pAYooMBAAjE0EdEiKI6AMQFcNKtsYMg== --osd-uuid 36f7b523-422c-4d1f-a7dd-37a53f244759 2026-02-25T22:05:38.943 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:38.940+0000 7f4c3fe63a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:38.944 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:38.940+0000 7f4c3fe63a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:38.945 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:38.940+0000 7f4c3fe63a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:38.946 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:38.940+0000 7f4c3fe63a40 -1 bdev(0x55cc385cc400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:38.946 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:38.940+0000 7f4c3fe63a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:05:39.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:05:39.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:39.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:05:39.639 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:05:39.639 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:39.753 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:05:39.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:05:39.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:39.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:39.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:39.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:39.768 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:39.764+0000 7ff303f82a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:39.769 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:39.764+0000 7ff303f82a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:39.770 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:39.768+0000 7ff303f82a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:39.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:05:39.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:39.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:05:39.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:39.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:39.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:39.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:39.869 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:39.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:39.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:39.982 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:40.143 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:40.140+0000 7ff303f82a40 -1 Falling back to public interface 2026-02-25T22:05:40.776 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:40.772+0000 7ff303f82a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:05:40.994 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:40.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:40.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:40.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:05:40.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:40.995 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:41.134 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:42.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:42.135 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:42.136 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:05:42.136 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:05:42.136 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:42.136 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:42.355 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/651634417,v1:127.0.0.1:6801/651634417] [v2:127.0.0.1:6802/651634417,v1:127.0.0.1:6803/651634417] exists,up 36f7b523-422c-4d1f-a7dd-37a53f244759 2026-02-25T22:05:42.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:42.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:42.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:42.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1558: test_is_clean: create_rbd_pool 2026-02-25T22:05:42.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:05:42.559 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:05:42.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:05:42.569 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:05:42.811 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:05:42.821 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:05:43.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:05:44.138 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: test_is_clean: wait_for_clean 2026-02-25T22:05:44.138 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:05:44.138 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:05:44.138 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:05:44.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:05:44.139 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:05:44.139 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:05:44.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:05:44.139 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:05:44.140 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:05:44.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:05:44.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:05:44.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:05:44.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:05:44.238 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:05:44.238 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:05:44.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:05:44.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:05:44.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:05:44.449 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:05:44.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:05:44.528 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:05:44.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:05:44.529 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:05:44.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:05:44.529 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:05:44.530 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:05:44.531 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:05:44.531 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:05:44.532 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:05:44.532 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:05:44.532 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:05:44.533 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:44.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:05:44.751 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:05:45.752 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:05:45.753 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:05:45.966 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:05:45.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:05:45.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:45.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:46.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:05:46.253 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:05:46.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:05:46.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:05:46.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:05:46.253 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:05:46.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:05:46.254 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:05:46.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:05:46.458 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:05:46.458 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:46.458 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:46.738 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:05:46.738 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:05:46.739 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:05:46.739 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1560: test_is_clean: is_clean 2026-02-25T22:05:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: get_num_pgs 2026-02-25T22:05:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:05:46.739 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:05:47.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: num_pgs=4 2026-02-25T22:05:47.016 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1547: is_clean: test 4 '!=' 0 2026-02-25T22:05:47.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: get_num_active_clean 2026-02-25T22:05:47.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:05:47.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:05:47.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:05:47.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:05:47.017 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:05:47.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: test 4 = 4 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: teardown td/ceph-helpers 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:47.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:47.332 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:47.332 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:47.333 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:47.334 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:47.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:47.335 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:47.335 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:47.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:47.336 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:47.337 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:47.337 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:47.338 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:47.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:47.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:47.344 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:47.345 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.345 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.345 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:47.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:47.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:47.347 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_kill_daemon td/ceph-helpers 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: test_kill_daemon: local dir=td/ceph-helpers 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:267: test_kill_daemon: setup td/ceph-helpers 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:47.348 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:47.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:47.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:47.349 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:47.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:47.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:47.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:47.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:47.352 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:47.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:47.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:47.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:47.353 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:47.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:47.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:47.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:47.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:47.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:47.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:47.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:47.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:05:47.359 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:05:47.360 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.360 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:05:47.361 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:05:47.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:05:47.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:05:47.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:47.383 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:05:47.383 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:47.384 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:47.384 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:47.384 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.384 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.384 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:47.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:47.414 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:05:47.415 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:05:47.415 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:47.415 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:47.415 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:05:47.416 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:05:47.416 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:47.417 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:05:47.471 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:05:47.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:47.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:47.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:05:47.472 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:05:47.473 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:47.473 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:47.473 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:47.473 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:47.473 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.474 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.474 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:47.474 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:47.474 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:05:47.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: test_kill_daemon: run_mgr td/ceph-helpers x 2026-02-25T22:05:47.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:05:47.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:05:47.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:05:47.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:05:47.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:05:47.527 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.637 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:47.638 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:47.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:47.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:270: test_kill_daemon: run_osd td/ceph-helpers 0 2026-02-25T22:05:47.660 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:47.661 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:47.662 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:47.663 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:05:47.664 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:47.664 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2ec91f23-53f6-421c-a4be-a0217637fff2 2026-02-25T22:05:47.664 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 2ec91f23-53f6-421c-a4be-a0217637fff2 2026-02-25T22:05:47.665 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2ec91f23-53f6-421c-a4be-a0217637fff2' 2026-02-25T22:05:47.665 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:47.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA7cp9pClItKBAA3OvygOxsHoAaS5Vkku5u/A== 2026-02-25T22:05:47.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA7cp9pClItKBAA3OvygOxsHoAaS5Vkku5u/A=="}' 2026-02-25T22:05:47.675 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2ec91f23-53f6-421c-a4be-a0217637fff2 -i td/ceph-helpers/0/new.json 2026-02-25T22:05:47.786 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:47.794 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:05:47.795 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7cp9pClItKBAA3OvygOxsHoAaS5Vkku5u/A== --osd-uuid 2ec91f23-53f6-421c-a4be-a0217637fff2 2026-02-25T22:05:47.811 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:47.808+0000 7f4376ccea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:47.813 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:47.808+0000 7f4376ccea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:47.814 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:47.812+0000 7f4376ccea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:47.814 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:47.812+0000 7f4376ccea40 -1 bdev(0x55f18d466400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:47.815 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:47.812+0000 7f4376ccea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:05:48.517 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:05:48.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:48.519 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:05:48.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:05:48.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:48.636 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:05:48.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:05:48.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:48.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:48.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:48.638 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:48.661 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:48.656+0000 7faab5bf3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:48.663 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:48.660+0000 7faab5bf3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:48.665 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:48.660+0000 7faab5bf3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:48.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:05:48.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:48.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:05:48.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:48.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:48.742 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:48.743 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:48.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:48.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:48.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:48.850 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:49.023 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:49.020+0000 7faab5bf3a40 -1 Falling back to public interface 2026-02-25T22:05:49.678 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:49.676+0000 7faab5bf3a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:05:49.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:49.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:49.851 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:05:49.852 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:05:49.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:49.855 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:49.986 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:50.987 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:05:50.987 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:50.988 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:05:50.988 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:05:50.988 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:50.989 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:51.209 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2086490816,v1:127.0.0.1:6801/2086490816] [v2:127.0.0.1:6802/2086490816,v1:127.0.0.1:6803/2086490816] exists,up 2ec91f23-53f6-421c-a4be-a0217637fff2 2026-02-25T22:05:51.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:05:51.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:05:51.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:05:51.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:272: test_kill_daemon: name_prefix=osd 2026-02-25T22:05:51.210 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: find td/ceph-helpers 2026-02-25T22:05:51.210 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: grep 'osd[^/]*\.pid' 2026-02-25T22:05:51.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-25T22:05:51.212 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:280: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2026-02-25T22:05:51.212 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-25T22:05:51.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=97930 2026-02-25T22:05:51.213 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=0 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local delays=1 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97930 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:51.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 1 2026-02-25T22:05:52.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97930 2026-02-25T22:05:52.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:52.214 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:52.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0 2026-02-25T22:05:52.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 1 2026-02-25T22:05:52.215 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:284: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2026-02-25T22:05:52.215 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=97930 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 97930 2026-02-25T22:05:52.217 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:52.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:52.218 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-25T22:05:52.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97930 2026-02-25T22:05:52.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:52.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:52.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2026-02-25T22:05:52.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.519 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97930 2026-02-25T22:05:52.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-25T22:05:52.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-25T22:05:52.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-25T22:05:52.520 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:287: test_kill_daemon: name_prefix=mgr 2026-02-25T22:05:52.520 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: find td/ceph-helpers 2026-02-25T22:05:52.520 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: grep 'mgr[^/]*\.pid' 2026-02-25T22:05:52.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-25T22:05:52.523 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:292: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2026-02-25T22:05:52.523 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2026-02-25T22:05:52.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=97074 2026-02-25T22:05:52.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-25T22:05:52.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-25T22:05:52.525 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:05:52.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 97074 2026-02-25T22:05:52.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:52.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:52.526 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-25T22:05:52.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97074 2026-02-25T22:05:52.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-25T22:05:52.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-25T22:05:52.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-25T22:05:52.627 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:295: test_kill_daemon: name_prefix=mon 2026-02-25T22:05:52.628 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: find td/ceph-helpers 2026-02-25T22:05:52.628 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: grep 'mon[^/]*\.pid' 2026-02-25T22:05:52.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-25T22:05:52.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:300: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2026-02-25T22:05:52.630 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mon.a.pid 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=96989 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 96989 2026-02-25T22:05:52.631 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:52.632 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:52.632 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-25T22:05:52.732 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.732 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 96989 2026-02-25T22:05:52.732 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:05:52.732 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:05:52.732 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2026-02-25T22:05:52.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:05:52.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 96989 2026-02-25T22:05:52.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-25T22:05:52.933 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-25T22:05:52.934 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-25T22:05:52.934 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: timeout 5 ceph status 2026-02-25T22:05:57.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: teardown td/ceph-helpers 2026-02-25T22:05:57.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:57.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:57.941 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:57.942 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:57.942 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:57.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:57.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:57.942 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:57.947 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:57.947 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:57.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:57.949 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:57.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:57.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:57.950 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:57.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:57.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:57.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:57.952 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:57.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:57.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:57.955 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:57.959 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:57.959 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:57.960 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:57.960 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:57.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:57.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:57.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:05:57.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_kill_daemons td/ceph-helpers 2026-02-25T22:05:57.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:357: test_kill_daemons: local dir=td/ceph-helpers 2026-02-25T22:05:57.962 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:358: test_kill_daemons: setup td/ceph-helpers 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:05:57.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:05:57.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:05:57.964 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:05:57.965 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:05:57.966 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:05:57.967 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:05:57.967 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:05:57.967 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:05:57.968 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:57.968 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:05:57.969 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:05:57.969 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:05:57.970 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:05:57.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:05:57.971 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:05:57.972 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:05:57.972 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:57.972 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:57.972 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:05:57.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:05:57.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:05:57.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:05:57.975 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:05:57.975 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:57.975 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:57.975 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:05:57.977 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:359: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:05:57.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:05:57.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:05:57.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:05:57.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:58.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:05:58.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:58.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:58.002 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:58.002 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:58.002 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:58.003 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:58.003 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:05:58.032 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:05:58.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:05:58.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:58.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:58.033 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:05:58.034 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:05:58.034 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:58.034 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:58.034 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:58.035 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:58.035 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:58.035 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:58.035 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:58.035 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:58.035 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:05:58.093 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:05:58.093 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:05:58.094 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:05:58.094 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:05:58.094 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:05:58.095 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:05:58.096 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:360: test_kill_daemons: run_mgr td/ceph-helpers x 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:05:58.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:05:58.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:05:58.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:58.265 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:58.266 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:58.266 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:58.266 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:58.266 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:58.267 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:58.267 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:05:58.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:361: test_kill_daemons: run_osd td/ceph-helpers 0 2026-02-25T22:05:58.287 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:05:58.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:05:58.289 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:05:58.290 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:05:58.291 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:05:58.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=96a21404-d836-41d3-b494-437077548f28 2026-02-25T22:05:58.291 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 96a21404-d836-41d3-b494-437077548f28' 2026-02-25T22:05:58.291 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 96a21404-d836-41d3-b494-437077548f28 2026-02-25T22:05:58.292 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:05:58.300 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBGcp9pQZvgERAARX5T4fYO+5MoB2v0em4V2Q== 2026-02-25T22:05:58.300 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBGcp9pQZvgERAARX5T4fYO+5MoB2v0em4V2Q=="}' 2026-02-25T22:05:58.301 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 96a21404-d836-41d3-b494-437077548f28 -i td/ceph-helpers/0/new.json 2026-02-25T22:05:58.412 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:58.420 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:05:58.421 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGcp9pQZvgERAARX5T4fYO+5MoB2v0em4V2Q== --osd-uuid 96a21404-d836-41d3-b494-437077548f28 2026-02-25T22:05:58.436 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:58.432+0000 7f8e8ffc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:58.437 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:58.432+0000 7f8e8ffc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:58.438 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:58.436+0000 7f8e8ffc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:58.439 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:58.436+0000 7f8e8ffc1a40 -1 bdev(0x561c6bfea400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:05:58.439 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:58.436+0000 7f8e8ffc1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:05:59.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:05:59.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:05:59.129 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:05:59.130 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:05:59.130 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:05:59.242 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:05:59.242 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:05:59.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:05:59.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:05:59.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:05:59.243 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:05:59.258 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:59.256+0000 7fc673e0ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:59.259 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:59.256+0000 7fc673e0ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:59.260 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:59.256+0000 7fc673e0ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:05:59.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:05:59.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:05:59.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:05:59.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:05:59.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:05:59.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:05:59.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:05:59.351 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:05:59.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:05:59.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:05:59.455 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:05:59.647 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:05:59.644+0000 7fc673e0ca40 -1 Falling back to public interface 2026-02-25T22:06:00.456 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:00.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:00.457 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:06:00.457 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:06:00.458 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:00.461 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:00.595 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:00.655 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:00.652+0000 7fc673e0ca40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:06:01.596 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:01.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:01.597 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:06:01.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:06:01.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:01.598 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:01.807 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3151514976,v1:127.0.0.1:6801/3151514976] [v2:127.0.0.1:6802/3151514976,v1:127.0.0.1:6803/3151514976] exists,up 96a21404-d836-41d3-b494-437077548f28 2026-02-25T22:06:01.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:06:01.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:06:01.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:06:01.808 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2026-02-25T22:06:01.809 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:01.809 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:01.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:01.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:01.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:02.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 1 2026-02-25T22:06:02.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:372: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:06:02.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:02.818 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:02.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:02.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:02.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:03.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:03.126 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:376: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2026-02-25T22:06:03.127 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:03.127 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:03.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:03.127 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:03.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:03.235 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:03.235 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:380: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2026-02-25T22:06:03.236 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:03.236 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:03.237 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:03.237 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:03.237 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:03.343 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:03.343 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: timeout 5 ceph status 2026-02-25T22:06:08.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:382: test_kill_daemons: teardown td/ceph-helpers 2026-02-25T22:06:08.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:06:08.349 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:06:08.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:06:08.350 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:08.351 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:08.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:08.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:08.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:08.355 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:08.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:06:08.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:06:08.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:06:08.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:06:08.359 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:06:08.359 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:06:08.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:08.360 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:06:08.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:06:08.361 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:08.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:06:08.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:06:08.363 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:06:08.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:06:08.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.366 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.366 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:06:08.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:06:08.368 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_objectstore_tool td/ceph-helpers 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: test_objectstore_tool: local dir=td/ceph-helpers 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: test_objectstore_tool: setup td/ceph-helpers 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:08.369 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:08.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:08.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:08.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:08.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:06:08.371 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:06:08.372 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:06:08.372 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:06:08.372 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:06:08.373 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:06:08.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:08.373 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:06:08.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:06:08.374 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:08.375 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:06:08.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:06:08.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:06:08.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:06:08.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.378 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:06:08.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:06:08.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:06:08.379 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:06:08.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:06:08.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:06:08.382 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:06:08.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:06:08.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:06:08.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:06:08.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:08.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:06:08.383 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:06:08.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:06:08.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:06:08.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:06:08.384 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:08.406 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:06:08.406 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:08.406 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:08.407 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:08.407 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.408 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.408 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:08.408 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:08.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:06:08.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:06:08.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:06:08.440 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:06:08.441 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.442 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:06:08.443 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:06:08.443 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:06:08.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:06:08.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:06:08.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:06:08.498 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:06:08.499 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:06:08.499 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:06:08.499 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:06:08.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: test_objectstore_tool: local osd=0 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: test_objectstore_tool: run_mgr td/ceph-helpers x 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:06:08.555 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:06:08.556 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:06:08.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:06:08.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:08.670 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:08.671 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:08.671 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.671 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.671 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:08.671 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:06:08.672 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:06:08.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: test_objectstore_tool: run_osd td/ceph-helpers 0 2026-02-25T22:06:08.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:06:08.692 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:06:08.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:06:08.694 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:06:08.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=05ece274-469d-417e-8190-a976079da825 2026-02-25T22:06:08.695 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 05ece274-469d-417e-8190-a976079da825' 2026-02-25T22:06:08.695 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 05ece274-469d-417e-8190-a976079da825 2026-02-25T22:06:08.695 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:06:08.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQcp9pYWDvKRAAA+wMSrHgMTjhhdfekj5NBA== 2026-02-25T22:06:08.704 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQcp9pYWDvKRAAA+wMSrHgMTjhhdfekj5NBA=="}' 2026-02-25T22:06:08.705 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 05ece274-469d-417e-8190-a976079da825 -i td/ceph-helpers/0/new.json 2026-02-25T22:06:08.814 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:06:08.822 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:06:08.824 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQcp9pYWDvKRAAA+wMSrHgMTjhhdfekj5NBA== --osd-uuid 05ece274-469d-417e-8190-a976079da825 2026-02-25T22:06:08.839 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:08.836+0000 7f0e4135ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:08.841 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:08.836+0000 7f0e4135ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:08.842 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:08.840+0000 7f0e4135ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:08.842 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:08.840+0000 7f0e4135ea40 -1 bdev(0x564499c0c400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:06:08.842 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:08.840+0000 7f0e4135ea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:06:09.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:06:09.597 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:06:09.599 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:06:09.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:06:09.599 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:06:09.724 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:06:09.725 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:06:09.725 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:06:09.726 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:06:09.726 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:06:09.726 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:06:09.747 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:09.744+0000 7f3cb9020a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:09.749 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:09.744+0000 7f3cb9020a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:09.750 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:09.748+0000 7f3cb9020a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:09.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:06:09.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:06:09.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:06:09.838 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:06:09.839 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:06:09.839 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:09.839 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:06:09.839 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:06:09.840 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:09.840 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:09.948 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:10.163 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:10.160+0000 7f3cb9020a40 -1 Falling back to public interface 2026-02-25T22:06:10.764 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:10.760+0000 7f3cb9020a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:06:10.949 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:10.949 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:10.949 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:06:10.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:06:10.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:10.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:11.107 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:12.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:12.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:12.108 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:06:12.109 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:06:12.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:12.109 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:12.326 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2680474410,v1:127.0.0.1:6801/2680474410] [v2:127.0.0.1:6802/2680474410,v1:127.0.0.1:6803/2680474410] exists,up 05ece274-469d-417e-8190-a976079da825 2026-02-25T22:06:12.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:06:12.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:06:12.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:06:12.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: test_objectstore_tool: create_rbd_pool 2026-02-25T22:06:12.327 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:06:12.530 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:06:12.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:06:12.541 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:06:12.769 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:06:12.779 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:06:13.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:06:14.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: test_objectstore_tool: wait_for_clean 2026-02-25T22:06:14.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:06:14.052 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:06:14.053 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:06:14.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:06:14.054 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:14.054 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:14.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:14.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:14.054 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:14.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:06:14.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:06:14.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:06:14.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:06:14.151 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:06:14.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:14.361 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:14.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:14.362 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:14.362 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:14.443 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:06:14.443 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:06:14.443 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:06:14.443 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:06:14.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:06:14.444 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:06:14.445 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:06:14.445 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:06:14.446 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:06:14.447 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:06:14.447 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:06:14.448 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:06:14.448 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:14.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:06:14.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:15.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:06:15.660 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:15.875 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:06:15.875 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:06:15.876 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:15.876 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:16.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:06:16.150 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:06:16.150 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:06:16.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:06:16.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:06:16.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:06:16.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:06:16.151 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:06:16.356 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:06:16.356 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:06:16.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:16.357 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:16.628 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:06:16.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:06:16.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:06:16.629 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:06:16.650 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: diff - /etc/group 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:16.651 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-25T22:06:16.958 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:06:17.128 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:17.129 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:17.131 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:17.131 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:17.131 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:17.131 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:17.131 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:06:17.132 INFO:tasks.workunit.client.0.trial163.stderr:start osd.0 2026-02-25T22:06:17.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:06:17.133 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:06:17.133 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:06:17.134 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:06:17.134 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:06:17.134 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:06:17.149 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:17.144+0000 7f8187a99a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:17.151 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:17.148+0000 7f8187a99a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:17.152 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:17.148+0000 7f8187a99a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:17.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:06:17.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:06:17.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:06:17.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:06:17.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:06:17.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:17.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:06:17.340 INFO:tasks.workunit.client.0.trial163.stderr:0 2026-02-25T22:06:17.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:17.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:17.556 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:17.595 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:17.592+0000 7f8187a99a40 -1 Falling back to public interface 2026-02-25T22:06:18.174 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:18.172+0000 7f8187a99a40 -1 osd.0 11 log_to_monitors true 2026-02-25T22:06:18.557 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:18.557 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:18.557 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:06:18.557 INFO:tasks.workunit.client.0.trial163.stderr:1 2026-02-25T22:06:18.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:18.558 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:18.780 INFO:tasks.workunit.client.0.trial163.stderr:osd.0 up in weight 1 up_from 14 up_thru 14 down_at 12 last_clean_interval [5,11) [v2:127.0.0.1:6800/3035996922,v1:127.0.0.1:6801/3035996922] [v2:127.0.0.1:6802/3035996922,v1:127.0.0.1:6803/3035996922] exists,up 05ece274-469d-417e-8190-a976079da825 2026-02-25T22:06:18.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:06:18.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:06:18.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:06:18.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:06:18.780 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:06:18.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:06:18.781 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:06:18.781 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:06:18.782 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:18.782 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:18.782 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:18.782 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:18.782 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:18.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:06:18.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:06:18.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:06:18.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:06:18.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:06:18.868 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:19.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:19.084 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:19.085 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:19.085 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:19.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-25T22:06:19.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-25T22:06:19.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-60129542147' 2026-02-25T22:06:19.163 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:06:19.163 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-60129542147 2026-02-25T22:06:19.163 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:06:19.164 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:06:19.165 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-60129542147 2026-02-25T22:06:19.165 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:06:19.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542147 2026-02-25T22:06:19.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 60129542147' 2026-02-25T22:06:19.167 INFO:tasks.workunit.client.0.trial163.stderr:waiting osd.0 seq 60129542147 2026-02-25T22:06:19.167 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:19.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 1 -lt 60129542147 2026-02-25T22:06:19.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:20.381 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:06:20.382 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:20.589 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 1 -lt 60129542147 2026-02-25T22:06:20.589 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:21.590 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:06:21.591 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:21.809 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-25T22:06:21.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:06:21.810 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:21.811 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:22.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:06:22.087 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:06:22.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:06:22.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:06:22.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:06:22.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:06:22.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:06:22.088 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:06:22.297 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:06:22.297 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:06:22.298 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:22.298 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:22.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:06:22.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:06:22.572 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:06:22.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-25T22:06:22.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-25T22:06:22.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:06:22.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-25T22:06:22.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:06:22.574 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:22.575 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-25T22:06:22.882 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2026-02-25T22:06:22.968 INFO:tasks.workunit.client.0.trial163.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2026-02-25T22:06:23.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: return 1 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: return 1 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: teardown td/ceph-helpers 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:23.056 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:23.162 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:23.163 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:06:23.164 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:06:23.164 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:06:23.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:06:23.166 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:06:23.166 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:06:23.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:23.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:06:23.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:06:23.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:23.168 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:06:23.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:06:23.170 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:06:23.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:06:23.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.176 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.176 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:06:23.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:06:23.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:06:23.178 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_pg_schedule_scrub td/ceph-helpers 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: test_pg_schedule_scrub: local dir=td/ceph-helpers 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: test_pg_schedule_scrub: setup td/ceph-helpers 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:23.179 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:23.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:23.180 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:23.180 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:06:23.181 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:06:23.181 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:06:23.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:06:23.182 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:06:23.183 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:06:23.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:23.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:06:23.184 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:06:23.184 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:23.185 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:06:23.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:06:23.187 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:06:23.188 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:06:23.188 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.188 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:06:23.189 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:06:23.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:06:23.190 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:06:23.191 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:06:23.191 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.191 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.191 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:06:23.193 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:06:23.194 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:06:23.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:06:23.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:06:23.195 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:23.215 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:06:23.215 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:23.215 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:23.215 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:23.216 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.216 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.216 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:23.216 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:23.244 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:06:23.245 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:06:23.245 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:06:23.245 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:06:23.245 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:06:23.246 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:06:23.246 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:06:23.246 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:06:23.247 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:06:23.302 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:06:23.302 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:06:23.302 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:06:23.302 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:06:23.302 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:06:23.304 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:06:23.357 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_pg_schedule_scrub: run_mgr td/ceph-helpers x --mgr_stats_period=1 2026-02-25T22:06:23.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:06:23.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:06:23.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:06:23.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:06:23.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:06:23.358 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:06:23.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:06:23.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:23.469 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:23.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:23.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.470 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:23.470 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:06:23.471 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-25T22:06:23.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2026-02-25T22:06:23.492 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:06:23.493 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:06:23.494 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:06:23.495 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:06:23.496 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:06:23.496 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=197f7a09-0b0e-4c36-8487-4e731ea6fe47 2026-02-25T22:06:23.496 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 197f7a09-0b0e-4c36-8487-4e731ea6fe47' 2026-02-25T22:06:23.496 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 197f7a09-0b0e-4c36-8487-4e731ea6fe47 2026-02-25T22:06:23.497 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:06:23.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBfcp9pHRggHhAAIsMFYYt7VwN9ggiBCNHIiw== 2026-02-25T22:06:23.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBfcp9pHRggHhAAIsMFYYt7VwN9ggiBCNHIiw=="}' 2026-02-25T22:06:23.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 197f7a09-0b0e-4c36-8487-4e731ea6fe47 -i td/ceph-helpers/0/new.json 2026-02-25T22:06:23.616 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:06:23.623 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:06:23.624 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfcp9pHRggHhAAIsMFYYt7VwN9ggiBCNHIiw== --osd-uuid 197f7a09-0b0e-4c36-8487-4e731ea6fe47 2026-02-25T22:06:23.639 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:23.636+0000 7fac90020a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:23.641 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:23.636+0000 7fac90020a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:23.642 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:23.640+0000 7fac90020a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:23.642 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:23.640+0000 7fac90020a40 -1 bdev(0x55a33a7a0400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:06:23.643 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:23.640+0000 7fac90020a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:06:24.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:06:24.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:06:24.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:06:24.441 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:06:24.441 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:06:24.564 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:06:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:06:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:06:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:06:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:06:24.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:06:24.589 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:24.584+0000 7fa71b98ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:24.591 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:24.588+0000 7fa71b98ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:24.592 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:24.588+0000 7fa71b98ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:24.676 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:06:24.677 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:06:24.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:24.677 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:24.785 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:25.091 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:25.088+0000 7fa71b98ba40 -1 Falling back to public interface 2026-02-25T22:06:25.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:25.786 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:25.787 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:06:25.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:06:25.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:25.787 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:26.003 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:26.019 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:26.016+0000 7fa71b98ba40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:06:27.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:27.004 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:27.005 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:06:27.005 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:06:27.005 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:27.006 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:27.223 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4148475510,v1:127.0.0.1:6801/4148475510] [v2:127.0.0.1:6802/4148475510,v1:127.0.0.1:6803/4148475510] exists,up 197f7a09-0b0e-4c36-8487-4e731ea6fe47 2026-02-25T22:06:27.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:06:27.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:06:27.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:06:27.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_pg_schedule_scrub: create_rbd_pool 2026-02-25T22:06:27.225 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:06:27.428 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:06:27.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:06:27.438 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:06:27.682 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:06:27.691 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:06:28.693 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:06:28.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_pg_schedule_scrub: wait_for_clean 2026-02-25T22:06:28.973 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:06:28.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:06:28.974 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:06:28.974 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:06:28.975 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:28.975 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:28.975 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:28.975 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:28.975 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:29.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:06:29.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:06:29.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:06:29.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:06:29.073 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:06:29.073 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:29.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:29.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:29.288 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:29.289 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:29.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:06:29.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:06:29.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:06:29.370 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:06:29.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:06:29.371 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:06:29.372 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:06:29.372 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:06:29.373 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:06:29.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:06:29.374 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:06:29.374 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:06:29.375 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:29.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836485 2026-02-25T22:06:29.591 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:30.592 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:06:30.593 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:30.802 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-25T22:06:30.803 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:06:30.803 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:30.803 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:31.076 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:06:31.076 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:06:31.077 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:06:31.077 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:06:31.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:06:31.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:06:31.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:06:31.078 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:06:31.282 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:06:31.283 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:06:31.283 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:31.283 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:31.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:06:31.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:06:31.565 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: pg_schedule_scrub: local pgid=1.0 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: pg_schedule_scrub: wait_for_pg_clean 1.0 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:31.566 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:31.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:31.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:31.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-25T22:06:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-25T22:06:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-25T22:06:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-25T22:06:31.823 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:06:31.823 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:32.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:32.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:32.036 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:32.037 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:32.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-25T22:06:32.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-25T22:06:32.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836489' 2026-02-25T22:06:32.116 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:06:32.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836489 2026-02-25T22:06:32.117 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:06:32.117 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:06:32.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-25T22:06:32.118 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:06:32.120 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836489 2026-02-25T22:06:32.120 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836489 2026-02-25T22:06:32.120 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-25T22:06:32.121 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:32.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-25T22:06:32.339 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:33.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:06:33.340 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:33.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-25T22:06:33.560 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:34.561 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:06:34.561 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:34.770 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836489 2026-02-25T22:06:34.770 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-25T22:06:34.770 INFO:tasks.workunit.client.0.trial163.stdout:#---------- 1.0 loop 0 2026-02-25T22:06:34.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-25T22:06:34.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-25T22:06:34.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-25T22:06:34.771 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-25T22:06:34.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-25T22:06:34.772 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: pg_schedule_scrub: get_last_scrub_stamp 1.0 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:34.851 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:34.852 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:34.852 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:35.054 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: pg_schedule_scrub: local last_scrub=2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:35.055 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: pg_schedule_scrub: ceph tell 1.0 schedule-scrub 2026-02-25T22:06:35.128 INFO:tasks.workunit.client.0.trial163.stdout:{ 2026-02-25T22:06:35.128 INFO:tasks.workunit.client.0.trial163.stdout: "deep": false, 2026-02-25T22:06:35.129 INFO:tasks.workunit.client.0.trial163.stdout: "must": false, 2026-02-25T22:06:35.129 INFO:tasks.workunit.client.0.trial163.stdout: "stamp": "2026-02-23T10:06:35.128847+0000" 2026-02-25T22:06:35.129 INFO:tasks.workunit.client.0.trial163.stdout:} 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: pg_schedule_scrub: wait_for_scrub 1.0 2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:35.137 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:35.138 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:35.340 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:06:27.628022+0000 '>' 2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:35.341 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:36.342 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:36.342 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:36.342 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:36.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:36.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:36.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:36.343 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:36.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-23T10:06:35.128847+0000 '>' 2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:36.546 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:37.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:37.547 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:37.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:37.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:37.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:37.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:37.548 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:37.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-23T10:06:35.128847+0000 '>' 2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:37.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:38.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:38.754 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:38.755 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:38.755 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:38.755 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:38.756 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:38.756 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:38.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-23T10:06:35.128847+0000 '>' 2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:38.963 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:39.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:39.964 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:39.965 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:39.965 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:39.965 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:39.965 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:39.965 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:40.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:06:38.060500+0000 '>' 2026-02-25T22:06:27.628022+0000 2026-02-25T22:06:40.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-25T22:06:40.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_pg_schedule_scrub: teardown td/ceph-helpers 2026-02-25T22:06:40.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:06:40.174 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:06:40.175 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:06:40.175 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:40.175 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:40.175 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:40.175 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:40.175 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:40.483 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:40.484 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:06:40.485 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:06:40.486 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:06:40.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:06:40.487 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:06:40.488 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:06:40.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:40.489 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:06:40.490 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:06:40.490 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:40.490 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:06:40.491 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:06:40.491 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:06:40.496 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:06:40.496 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.496 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.497 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_pg_scrub td/ceph-helpers 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_scrub: local dir=td/ceph-helpers 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: test_pg_scrub: setup td/ceph-helpers 2026-02-25T22:06:40.499 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:40.500 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:40.501 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:40.501 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:06:40.502 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:06:40.503 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:06:40.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:06:40.504 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:06:40.505 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:06:40.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:40.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:06:40.506 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:06:40.506 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:06:40.507 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:06:40.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:06:40.508 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:06:40.509 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:06:40.510 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.510 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-25T22:06:40.510 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:06:40.511 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:06:40.511 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:06:40.512 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:06:40.512 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.512 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.512 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.56683 2026-02-25T22:06:40.514 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1944: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:06:40.515 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:06:40.516 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:40.537 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:06:40.537 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:40.537 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:40.538 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:40.538 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.538 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.538 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:40.538 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:06:40.566 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:06:40.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:06:40.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:06:40.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:06:40.567 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:06:40.568 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:06:40.569 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get fsid 2026-02-25T22:06:40.623 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:06:40.623 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:06:40.623 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:06:40.623 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:06:40.624 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.56683/ceph-mon.a.asok 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:06:40.625 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.56683/ceph-mon.a.asok config get mon_host 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: test_pg_scrub: run_mgr td/ceph-helpers x 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:06:40.679 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:06:40.790 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:06:40.790 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:40.790 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:40.791 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:40.791 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.791 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.791 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:40.792 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:06:40.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:06:40.815 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: test_pg_scrub: run_osd td/ceph-helpers 0 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:06:40.816 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:40.817 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:06:40.818 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:06:40.819 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:06:40.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c037d44-cb67-443e-bc66-f47db76097b2 2026-02-25T22:06:40.819 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6c037d44-cb67-443e-bc66-f47db76097b2' 2026-02-25T22:06:40.820 INFO:tasks.workunit.client.0.trial163.stdout:add osd0 6c037d44-cb67-443e-bc66-f47db76097b2 2026-02-25T22:06:40.820 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:06:40.829 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwcp9pOUVkMRAA1HGAQA5GJhCYPcbtb9XmUQ== 2026-02-25T22:06:40.829 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwcp9pOUVkMRAA1HGAQA5GJhCYPcbtb9XmUQ=="}' 2026-02-25T22:06:40.829 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c037d44-cb67-443e-bc66-f47db76097b2 -i td/ceph-helpers/0/new.json 2026-02-25T22:06:40.943 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:06:40.950 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:06:40.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwcp9pOUVkMRAA1HGAQA5GJhCYPcbtb9XmUQ== --osd-uuid 6c037d44-cb67-443e-bc66-f47db76097b2 2026-02-25T22:06:40.968 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:40.963+0000 7fc0b25dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:40.969 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:40.963+0000 7fc0b25dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:40.970 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:40.967+0000 7fc0b25dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:40.971 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:40.967+0000 7fc0b25dfa40 -1 bdev(0x561bdf8fa400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:06:40.971 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:40.967+0000 7fc0b25dfa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:06:41.635 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:06:41.635 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:06:41.636 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:06:41.637 INFO:tasks.workunit.client.0.trial163.stdout:adding osd0 key to auth repository 2026-02-25T22:06:41.637 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:06:41.752 INFO:tasks.workunit.client.0.trial163.stdout:start osd.0 2026-02-25T22:06:41.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:06:41.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=806c5fa0-46a5-4104-95dd-807af436ece2 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.56683/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:06:41.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:06:41.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:06:41.753 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:06:41.768 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:41.763+0000 7f7385f9fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:41.769 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:41.763+0000 7f7385f9fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:41.770 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:41.767+0000 7f7385f9fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:06:41.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:06:41.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:06:41.864 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:06:41.865 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:06:41.865 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:06:41.865 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:41.865 INFO:tasks.workunit.client.0.trial163.stdout:0 2026-02-25T22:06:41.866 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:06:41.866 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:41.866 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:41.976 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:42.207 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:42.203+0000 7f7385f9fa40 -1 Falling back to public interface 2026-02-25T22:06:42.827 INFO:tasks.workunit.client.0.trial163.stderr:2026-02-25T22:06:42.823+0000 7f7385f9fa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:06:42.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:42.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:42.978 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:06:42.978 INFO:tasks.workunit.client.0.trial163.stdout:1 2026-02-25T22:06:42.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:42.979 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:43.094 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:06:44.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:06:44.095 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:06:44.096 INFO:tasks.workunit.client.0.trial163.stdout:2 2026-02-25T22:06:44.096 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:06:44.096 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:06:44.097 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:06:44.303 INFO:tasks.workunit.client.0.trial163.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1896355548,v1:127.0.0.1:6801/1896355548] [v2:127.0.0.1:6802/1896355548,v1:127.0.0.1:6803/1896355548] exists,up 6c037d44-cb67-443e-bc66-f47db76097b2 2026-02-25T22:06:44.303 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:06:44.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:06:44.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:06:44.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_scrub: create_rbd_pool 2026-02-25T22:06:44.304 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:06:44.507 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' does not exist 2026-02-25T22:06:44.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:06:44.518 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:06:44.734 INFO:tasks.workunit.client.0.trial163.stderr:pool 'rbd' already exists 2026-02-25T22:06:44.743 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:06:45.744 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:06:46.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_pg_scrub: wait_for_clean 2026-02-25T22:06:46.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:06:46.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:06:46.024 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:06:46.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:06:46.025 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:46.025 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:46.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:46.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:46.025 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:46.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:06:46.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:06:46.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:06:46.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:06:46.132 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:06:46.133 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:46.350 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:46.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:46.351 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:46.351 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:46.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:06:46.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:06:46.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:06:46.432 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:06:46.433 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:06:46.433 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:06:46.434 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:06:46.434 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:06:46.434 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:06:46.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:06:46.435 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:06:46.436 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:06:46.436 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:46.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:06:46.658 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:47.659 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:06:47.660 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:47.876 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:06:47.876 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:06:47.877 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:47.877 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:48.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:06:48.167 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:06:48.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:06:48.168 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:06:48.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:06:48.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:06:48.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:06:48.169 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:06:48.380 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:06:48.380 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:06:48.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:06:48.381 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:06:48.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:06:48.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:06:48.656 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:06:48.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_scrub: pg_scrub 1.0 2026-02-25T22:06:48.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-25T22:06:48.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-25T22:06:48.657 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-25T22:06:48.657 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-25T22:06:48.658 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:48.658 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:48.658 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:48.658 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:48.658 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:48.920 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-25T22:06:48.920 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-25T22:06:48.920 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-25T22:06:48.920 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-25T22:06:48.920 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:06:48.920 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:49.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:49.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:49.143 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:49.143 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:49.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-25T22:06:49.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-25T22:06:49.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836486' 2026-02-25T22:06:49.223 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:06:49.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836486 2026-02-25T22:06:49.223 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:06:49.224 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:06:49.225 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-25T22:06:49.225 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:06:49.226 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836486 2026-02-25T22:06:49.227 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-25T22:06:49.227 INFO:tasks.workunit.client.0.trial163.stdout:waiting osd.0 seq 21474836486 2026-02-25T22:06:49.227 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:49.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836486 2026-02-25T22:06:49.439 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:50.440 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:06:50.441 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:50.647 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836486 2026-02-25T22:06:50.648 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:06:51.649 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:06:51.649 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:06:51.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-25T22:06:51.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-25T22:06:51.868 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-25T22:06:51.869 INFO:tasks.workunit.client.0.trial163.stdout:#---------- 1.0 loop 0 2026-02-25T22:06:51.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-25T22:06:51.869 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-25T22:06:51.870 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-25T22:06:51.870 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-25T22:06:51.870 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-25T22:06:51.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-25T22:06:51.952 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-25T22:06:51.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-25T22:06:51.953 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-25T22:06:51.953 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-25T22:06:51.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:51.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:51.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:51.954 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:52.158 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:52.159 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-25T22:06:52.366 INFO:tasks.workunit.client.0.trial163.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-25T22:06:52.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:52.376 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:52.377 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:52.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:52.378 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:52.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:06:44.680573+0000 '>' 2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:52.585 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:53.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:53.587 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:53.587 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:53.587 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:53.588 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:53.588 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:53.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:06:44.680573+0000 '>' 2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:53.792 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:54.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:54.793 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:54.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:54.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:54.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:54.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:54.794 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:54.998 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:06:44.680573+0000 '>' 2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:54.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:06:55.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:06:55.999 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:06:56.000 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:06:56.000 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:06:56.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:06:56.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:06:56.001 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:06:56.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:06:52.969044+0000 '>' 2026-02-25T22:06:44.680573+0000 2026-02-25T22:06:56.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-25T22:06:56.210 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-25T22:06:56.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:06:56.211 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:06:56.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:06:56.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:06:56.211 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: WAIT_FOR_CLEAN_TIMEOUT=10 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: TIMEOUT=2 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: pg_scrub 1.0 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-25T22:06:56.318 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-25T22:06:56.319 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 10 1 3 2026-02-25T22:06:56.319 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:06:56.319 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:06:56.320 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:06:56.320 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:06:56.320 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:06:56.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '1') 2026-02-25T22:06:56.353 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-25T22:06:56.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-25T22:06:56.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-25T22:06:56.354 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=2 2026-02-25T22:06:56.354 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:06:56.580 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:06:56.580 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:06:56.580 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:06:56.580 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 2 ceph tell osd.0 flush_pg_stats 2026-02-25T22:06:58.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T22:06:58.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T22:06:58.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T22:06:58.586 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T22:06:58.587 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-25T22:06:58.587 INFO:tasks.workunit.client.0.trial163.stdout:#---------- 1.0 loop 0 2026-02-25T22:06:58.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-25T22:06:58.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-25T22:06:58.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-25T22:06:58.588 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-25T22:06:58.588 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-25T22:06:58.588 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-26T01:03:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-26T01:03:21.829 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-26T01:03:21.829 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1: is_pg_clean: teardown td/ceph-helpers 1 2026-02-26T01:03:21.829 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-26T01:03:21.829 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2026-02-26T01:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-26T01:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-26T01:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-26T01:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-26T01:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-26T01:03:21.830 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-26T01:03:21.917 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-26T01:03:21.917 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-26T01:03:21.918 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-26T01:03:21.919 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-26T01:03:21.920 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-26T01:03:21.921 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-26T01:03:21.921 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-26T01:03:21.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T01:03:21.922 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-26T01:03:21.923 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-26T01:03:21.923 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T01:03:21.924 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-26T01:03:21.926 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2026-02-26T01:03:21.926 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2026-02-26T01:03:21.926 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2026-02-26T01:03:21.927 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2026-02-26T01:03:21.928 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-26T01:03:21.933 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-26T01:03:21.933 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-26T01:03:21.933 INFO:tasks.workunit.client.0.trial163.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.56683 2026-02-26T01:03:21.933 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.56683 2026-02-26T01:03:21.935 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-26T01:03:21.935 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-26T01:03:21.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-26T01:03:21.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 0 >= 5 )) 2026-02-26T01:03:21.935 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 1 2026-02-26T01:03:22.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-26T01:03:22.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-26T01:03:22.936 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 1' 2026-02-26T01:03:22.936 INFO:tasks.workunit.client.0.trial163.stdout:#---------- 1.0 loop 1 2026-02-26T01:03:22.937 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-26T01:03:22.937 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-26T01:03:22.937 INFO:tasks.workunit.client.0.trial163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-26T01:03:22.937 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-26T01:03:22.938 INFO:tasks.workunit.client.0.trial163.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-26T01:04:21.829 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2026-02-26T01:05:21.830 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2026-02-26T01:05:21.830 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2026-02-26T01:05:21.831 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-26T01:05:22.157 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_Ericmzhang_ceph_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa/tasks/workunit.py", line 144, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_Ericmzhang_ceph_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa/tasks/workunit.py", line 292, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_Ericmzhang_ceph_0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/qa/tasks/workunit.py", line 433, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on trial163 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2026-02-26T01:05:22.164 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T01:05:22.182 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T01:05:22.182 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T01:05:22.291 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T01:05:22.303 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T01:05:22.303 DEBUG:teuthology.orchestra.run.trial163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T01:05:22.447 INFO:teuthology.orchestra.run.trial163.stdout: remote refid st t when poll reach delay offset jitter 2026-02-26T01:05:22.447 INFO:teuthology.orchestra.run.trial163.stdout:============================================================================== 2026-02-26T01:05:22.447 INFO:teuthology.orchestra.run.trial163.stdout:*108.61.215.221 50.205.57.38 2 u 133 1024 377 18.913 -3.763 1.991 2026-02-26T01:05:22.447 INFO:teuthology.orchestra.run.trial163.stdout:+149.248.12.167 132.239.1.6 2 u 629 1024 377 63.103 -2.395 2.036 2026-02-26T01:05:22.447 INFO:teuthology.orchestra.run.trial163.stdout:+lax1.us.ntp.li 131.215.220.143 2 u 110 1024 377 65.735 -1.610 1.331 2026-02-26T01:05:22.447 INFO:teuthology.orchestra.run.trial163.stdout:+199.101.96.52 129.6.15.29 2 u 159 1024 377 7.869 -2.625 4.459 2026-02-26T01:05:22.448 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T01:05:22.460 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T01:05:22.460 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T01:05:22.477 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T01:05:22.491 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T01:05:22.507 INFO:teuthology.task.internal:Duration was 11548.717580 seconds 2026-02-26T01:05:22.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T01:05:22.522 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T01:05:22.522 DEBUG:teuthology.orchestra.run.trial163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T01:05:22.548 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T01:05:22.548 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial163.front.sepia.ceph.com 2026-02-26T01:05:22.549 DEBUG:teuthology.orchestra.run.trial163:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T01:05:22.595 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T01:05:22.595 DEBUG:teuthology.orchestra.run.trial163:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T01:05:22.864 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T01:05:22.864 DEBUG:teuthology.orchestra.run.trial163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T01:05:22.873 INFO:teuthology.orchestra.run.trial163.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T01:05:22.874 INFO:teuthology.orchestra.run.trial163.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T01:05:22.874 INFO:teuthology.orchestra.run.trial163.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T01:05:22.875 INFO:teuthology.orchestra.run.trial163.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T01:05:22.875 INFO:teuthology.orchestra.run.trial163.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T01:05:22.943 INFO:teuthology.orchestra.run.trial163.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T01:05:22.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T01:05:22.960 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T01:05:22.961 DEBUG:teuthology.orchestra.run.trial163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T01:05:22.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T01:05:23.012 DEBUG:teuthology.orchestra.run.trial163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T01:05:23.044 INFO:teuthology.orchestra.run.trial163.stdout:kernel.core_pattern = core 2026-02-26T01:05:23.054 DEBUG:teuthology.orchestra.run.trial163:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T01:05:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T01:05:23.100 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T01:05:23.117 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T01:05:23.121 DEBUG:teuthology.misc:Transferring archived files from trial163:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_21:45:26-rados:standalone-wip-mon-colocate-distro-default-trial/70636/remote/trial163 2026-02-26T01:05:23.122 DEBUG:teuthology.orchestra.run.trial163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T01:05:36.576 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T01:05:36.576 DEBUG:teuthology.orchestra.run.trial163:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T01:05:36.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T01:05:36.685 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T01:05:36.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T01:05:36.711 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T01:05:36.711 DEBUG:teuthology.orchestra.run.trial163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T01:05:36.715 INFO:teuthology.orchestra.run.trial163.stdout: 1311292 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 26 01:05 /home/ubuntu/cephtest 2026-02-26T01:05:36.716 INFO:teuthology.orchestra.run.trial163.stdout: 1321456 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 25 21:56 /home/ubuntu/cephtest/mnt.0 2026-02-26T01:05:36.716 INFO:teuthology.orchestra.run.trial163.stdout: 1321457 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 25 22:03 /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T01:05:36.716 INFO:teuthology.orchestra.run.trial163.stdout: 1333531 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 25 22:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-26T01:05:36.716 INFO:teuthology.orchestra.run.trial163.stdout: 1333534 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 26 01:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2026-02-26T01:05:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T01:05:36.717 INFO:teuthology.orchestra.run.trial163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2026-02-26T01:05:36.717 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial163 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-26T01:05:36.718 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T01:05:36.730 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T01:05:36.746 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial163 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-26T01:05:36.751 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 11548.717579841614 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on trial163 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_emzhang@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-26T01:05:36.751 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T01:05:36.866 INFO:teuthology.run:FAIL